How to specify scale_pos_weight value at runtime in Hyperopt?

M

megjosh

Guest
megjosh Asks: How to specify scale_pos_weight value at runtime in Hyperopt?
I want to use LighgbmClassifier for a binary Classification. for Hyper Parameter tuning I want to use Hyperopt. The Dataset is imbalanced. Using Sklearns class_weight.compute_class_weight as shown below

Code: 
        clas_wts_arr = class_weight.compute_class_weight('balanced',np.unique(y_trn),y_trn)   
        self.scale_pos_wt = clas_wts_arr[0] / clas_wts_arr[1]

The following is the space parameter that I am passing to the objective function

Code: 
        space = {'objective' : hp.choice('objective', objective_list),
                 'boosting' : hp.choice('boosting', boosting_list),
                 'metric' : hp.choice('metric', metric_list),
                 "max_depth": hp.quniform("max_depth", 1, 15,2),
                 'min_data_in_leaf': hp.quniform('min_data_in_leaf', 1, 256, 1),                     
                 'num_leaves': hp.quniform('num_leaves', 7, 150, 1),
                 'feature_fraction' : hp.quniform('feature_fraction', 0.5, 1, 0.01),
                 'min_gain_to_split' : hp.quniform('min_gain_to_split', 0.1, 5, 0.01),
                 'lambda_l1' : hp.uniform('lambda_l1', 0, 5),
                 'lambda_l2' : hp.uniform('lambda_l2', 0, 5),
                 'feature_pre_filter': False}

My question will the following set scale_pos_weight properly in the space dictionary

Code: 
        #set scale pos weight explicitly
        space['scale_pos_weight'] = self.scale_pos_wt

If that is wrong then what would be the correct way to set scale_pos_weight at runtime in the space dictionary that is passed to the Objective fn that is in turn passed to the fmin of Hyperopt.

Thanks for your help and answers.

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.
Click to expand...
 
You must log in or register to reply here.

Unreplied Threads

Provide me an essay about the Data Analyst job placements offered by Meghnad Desai Academy Of Economics

Sanjit Ku Asks: Provide me an essay about the Data Analyst job placements offered by Meghnad Desai Academy Of Economics
Brief details required I want to pursue data science course.

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.
Click to expand...

Training a model that maps embedding from (image, text) to text

Glinty Asks: Training a model that maps embedding from (image, text) to text
I have created embedding say A which is created my concatenating embedding of image and embedding of text, that is concat(img_embedding,text_embedding). Now, I have pairs such as (img_embedding,text_embedding) maps to (another text_embedding).
Like a pair ((img_embedding,text_embedding),another text embedding)
So, I want to train model that will take (img_embedding, text_embedding) and map to corresponding text embedding.
In sentence Transformers, we can train text to text mapping using MultipleNegativesRankingLoss. https://www.sbert.net/docs/package_reference/losses.html#multiplenegativesrankingloss
What are the approaches train model that can do (img_embedding,text_embedding) mapping to another text_embedding?

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.
Click to expand...

Logistic Regression for prediction

Ledian K. Asks: Logistic Regression for prediction
I would like to ask about the theoretical approach of using Logistic Regression for customer data and more specifically Churn Prediction (in BigQuery and Python).

I have my customer data for an online shop and I would like to predict if the customer will churn based on some characteristics. I have created my dataset and the Churn label (based on the hypothesis that if the customer hasn't bought something in the last year then it is assumed that the customer is churned since we are dealing with a non-contractual setting).

I am using 3 years of data (2019-2021), which includes ~3M customers and 43 features, and as I said, a customer is considered to be churned if the customer didn't place an order in 2021.

  1. I checked the distribution of my label which is ~balanced.
  2. I checked for some Logistic Regression assumptions such as multicollinearity, outlier influence etc.
  3. I split the data into 80% training data, 10% evaluation data, 10% prediction data.
  4. I checked the model's performance by looking at the classification metrics (Accuracy, Recall etc.)

My question would be:

We have the predictions of the 10% of the data (i.e. the probabilities that a customer will churn). Could we have the probabilities for all the other customers that belong in the training dataset and in the evaluation dataset?

In other words, what would be the next steps after we have trained and have checked that we could use the model, if your final goal would be to have in the end the probabilities of your customers to churn or to not churn?

Thank you in advance for your help!

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.
Click to expand...

The Height of subtree in Heap

Ramon Fonis Asks: The Height of subtree in Heap
In order to find the recurrence function of The Height in Heap, the following figure is drawn.

enter image description here

Question 1: How can we compute the height if Right subtree in form of Log in base 3, and why do we have the height of Left subtree as log in base 3/2?

Question 2: it is stated that:
"for a complete binary tree to have the maximum height the last level should be half full."
i don't grasp the logic behind this line.why?

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.
Click to expand...

[Solved] Warning using exact_extract in R

OgeiD Asks: Warning using exact_extract in R
I have a grid cell shapefile of Africa of 55 km x 55 km.

I would like to calculate a weighted average of the Terrain Ruggedness Index (explain here https://diegopuga.org/data/rugged/#grid) using as a weights the values of the area of each cell. To do that, I have two rasters, one with the Terrain Ruggedness Index, and another with the area of each cell. Both of them have a resolution of 30 by 30 arcsecond. As the authors of the link explained, it's important to take into account that the sea-level surface varies in proportion to the cosine of its latitude, this is why I weights by the area of the cell.

I use the following code:

Code: 
library(raster)
library(sf)
library(exactextractr)


shape$ruggedness = exact_extract(tri, shape, fun = "weighted_mean", weights = cells)

where shape is my grid cell, and tri and cells are the raster data of the ruggedness index and area of the cell respectively.

When I run the code, the following appears: "Cannot preload entire working area of 109568941 cells with max_cells_in_memory = 3e+07. Raster values will be read for each feature individually."

I do not understand why that is produced and if this can affect my results.

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your response here to help other visitors like you. Thank you, solveforum.
Click to expand...

[Solved] Problem with gdal.WarpOptions concerning reference

  • Nikolay Yasinskiy
  • Geography
  • Replies: 0
Nikolay Yasinskiy Asks: Problem with gdal.WarpOptions concerning reference
I have a tiff file that I want to crop down to a shp cutline and reproject.

If the code is:

Code: 
clip_spec=gdal.WarpOptions(format='GTiff',cutlineDSName=os.getcwd()+'/moksha/DEM/cutline2.shp',cropToCutline=True,dstNodata=-9999)
cutted=gdal.Warp(os.getcwd()+'/moksha/DEM/cutted22_m.tif',src,options=clip_spec,srcSRS='EPSG:4326',dstSRS='EPSG:32638')

I get a file with nodata.

If I include srs to options as:

Code: 
clip_spec=gdal.WarpOptions(format='GTiff',cutlineDSName=os.getcwd()+'/moksha/DEM/cutline2.shp',cropToCutline=True,dstNodata=-9999,srcSRS='EPSG:4326',dstSRS='EPSG:32638')
cutted=gdal.Warp(os.getcwd()+'/moksha/DEM/cutted22_m.tif',src,options=clip_spec)

I get - TypeError: in method 'wrapper_GDALWarpDestName', argument 4 of type 'GDALWarpAppOptions *

Anywhere I looked on web - these options are treated correctly. gdalTranslate worked well for reprojecting, but I would like to use gdal.Warp, as I have to crop the file. Tried to type otherwise with **kwargs - same thing. What could be the problem here and how can I make it work finally?

Files are available through the link

UPD: The problem was only in Jupiter, commandline works nice.

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your response here to help other visitors like you. Thank you, solveforum.
Click to expand...

Oxidation of MeOH in Metanolated water to reduce MeOH from 500ppm to almost nil in waste water treatment

yousef Asks: Oxidation of MeOH in Metanolated water to reduce MeOH from 500ppm to almost nil in waste water treatment
we have problem to dump our WW to the evaporation pound due to environmental regulation and need to reduce the MeOH in ww to maximum 3ppm. actually the only system that we have is stripper column which is not much effective. we tried acetic Acid to reduce the MeOH in Lab but the result is not satisfaction. is there any other solution except biological solution?

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.
Click to expand...

Latest posts

Newest Members

Top