I'm trying to deploy a locally trained RandomForest model into Azure Machine Learning Studio.

training code (whentrain.ipynb) :

#import libs and packages
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn.metrics import r2_score
from math import sqrt
from sklearn.ensemble import RandomForestRegressor
from sklearn.preprocessing import LabelEncoder
from imblearn.over_sampling import SMOTE
import xgboost as xgb
from sklearn.metrics import accuracy_score
from azureml.core import Workspace, Dataset
# get existing workspace
workspace = Workspace.from_config(path="config.json")
# get the datastore to upload prepared data
datastore = workspace.get_default_datastore()
# load the dataset which is placed in the data folder
dataset = Dataset.Tabular.from_delimited_files(datastore.path('UI/12-23-2021_023530_UTC/prepped_data101121.csv'))
dataset = dataset.to_pandas_dataframe()
# Create the outputs directories to save the model and images
os.makedirs('outputs/model', exist_ok=True)
os.makedirs('outputs/output', exist_ok=True)
dataset['Date'] = pd.to_datetime(dataset['Date'])
dataset = dataset.set_index('Date')
scaler = MinMaxScaler()
#inputs
X = dataset.iloc[:, 1:]
#output
y = dataset.iloc[:, :1]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state= 42, shuffle=True)
X_train = scaler.fit_transform(X_train)
X_test = scaler.fit_transform(X_test)
model1 = RandomForestRegressor(n_estimators = 6,
                                   max_depth = 10,
                                   min_samples_leaf= 1,
                                   oob_score = 'True',
                                   random_state=42)
model1.fit(X_train, y_train.values.ravel())
y_pred2 = model1.predict(X_test)

And here is the code on the estimator part (estimator.ipynb):

from azureml.core import Experiment
from azureml.core import Workspace
from azureml.core.compute import ComputeTarget, AmlCompute
from azureml.core.compute_target import ComputeTargetException
from azureml.train.dnn import TensorFlow
from azureml.widgets import RunDetails
import os
workspace = Workspace.from_config(path="config.json")
exp = Experiment(workspace=workspace, name='azure-exp')
cluster_name = "gpucluster"
    compute_target = ComputeTarget(workspace=workspace, name=cluster_name)
    print('Found existing compute target')
except ComputeTargetException:
    print('Creating a new compute target...')
    compute_config = AmlCompute.provisioning_configuration(vm_size='Standard_DS3_v2',
                                                           max_nodes=1)
    compute_target = ComputeTarget.create(workspace, cluster_name, compute_config)
    compute_target.wait_for_completion(show_output=True)  # , min_node_count=None, timeout_in_minutes=20)
    # For a more detailed view of current AmlCompute status, use get_status()
    print(compute_target.get_status().serialize())
from azureml.core import ScriptRunConfig
source_directory = os.getcwd()
from azureml.core import Environment
myenv = Environment("user-managed-env")
myenv.python.user_managed_dependencies =True
from azureml.core import Dataset
test_data_ds = Dataset.get_by_name(workspace, name='prepped_data101121')
src = ScriptRunConfig(source_directory=source_directory,
                      script='whentrain.ipynb',
                      arguments=['--input-data', test_data_ds.as_named_input('prepped_data101121')],
                      compute_target=compute_target,
                      environment=myenv)
run = exp.submit(src)
RunDetails(run).show()
run.wait_for_completion(show_output=True)

The error that happens in run.wait_for_completion states :

[stderr]Traceback (most recent call last):
[stderr]  File "whentrain.ipynb", line 107, in <module>
[stderr]    "notebookHasBeenCompleted": true
[stderr]NameError: name 'true' is not defined
[stderr]

As you can see in my whentrain.ipynb, it does not even reach line 107, and I could not find where this error come from. So how do I fix it?

I'm running the Notebook on Python 3.

UPDATE:

Okay, after a little adjustment that should not affect the whole code (I just removed some extra columns, added model save code in whentrain.ipynb making use of import os) it's now giving me somewhat the same error.

[stderr]Traceback (most recent call last):
[stderr]  File "whentrain.ipynb", line 115, in <module>
[stderr]    "source_hidden": false,
[stderr]NameError: name 'false' is not defined
[stderr]
											

@Ash I believe the field "source_hidden" is used by the notebook metadata for the cells that are used in your notebook. If you try to download the notebook and open it in an editor you can see this field. I am not sure why this error is flagged, have you downloaded the notebook from another source or are you using one cell for all the code?

I would suggest to open a new blank notebook in the portal and then use different cells for each step so you can isolate any issue with the code you are using. I hope this helps!!

Thank you for the reply! Well, whentrain.ipynb is in another ipynb file, and estimator is in another ipynb file. whentrain.ipynb is working fine when I run it alone, even tried using the model trained to predict a values. However it the error happened here in estimator.ipynb.

 run.wait_for_completion(show_output=True)  

as if saying that the whentrain.ipynb file has an issue, but it is working fine alone.

Does that mean that the error comes from my notebook metadata? I wonder if there's a way for me to fix it there. It's as if there is some library I am not importing.

Here's my file directories by the way

@Ash Ok, I think the issue is here.

src = ScriptRunConfig(source_directory=source_directory,  
                       script='whentrain.ipynb',  
                       arguments=['--input-data', test_data_ds.as_named_input('prepped_data101121')],  
                       compute_target=compute_target,  
                       environment=myenv)  

The script parameter is set to the notebook "whentrain.ipynb", This should be a python script *.py which can train your model. Since you are using the notebook filename the entire source of jupyter notebook is loaded and it fails with these errors. You can lookup samples on azure ml notebook github repo for reference. I think if you can convert your whentrain.ipynb file to a python script whentrain.py and save it the current folder structure you should be able to use it in this step.

The error in this case is because the estimator or scriptConfig is running your training script on your compute target that is passed as compute_target parameter and not on the local notebook compute instance. As part of the dependencies for your environment you need to add them to ensure they are available.

myenv = Environment("myenv")
myenv.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn', 'packaging'])
You are right, I was missing the dependencies!    

Thank you so much for being a great help. I have more issues past ScriptRunConfig though, particularly on model deployment. I'm trying to upload it to Power BI and I've been following the tutorials from Microsoft Documents, but its been confusing. Is there any way I could contact you or do I open up a new question in the forum? (Very new to this, my apologies!)

Great. You can open a new thread or post an issue on the docs repo from which you are following for clarification. This helps other community members to chip in and provide guidance if they have faced similar issues. Most of the customers open support cases to get 1:1 help from service engineers if any of these posts do not help. Thanks.