HP HPE2-N69 - Using HPE AI and Machine Learning

Pool 1 has two agents, each with 100 aux containers, and a total of 0 slots. This means that the cluster is configured to run CPU-only workloads, such as running a CPU-only Jupyter Notebook. Training, GPU Jupyter Notebook, and validation workloads cannot be run on this cluster due to the lack of GPU resources.

Trial 3 is scheduled on 8 of the slots. Then, after Trial 1 has finished, it receives 16 more slots. This is because priority scheduling is used in the HPE Machine Learning Development Environment resource pool, which means higher priority tasks will be given priority over lower priority tasks. As such, Trial 3 with priority 1 will be given priority over Trial 2 with priority 50.

HPE Machine Learning Development Environment uses Amazon S3 to store checkpoints. To find the location of the best checkpoint created during an experiment, you need to look for a "determined-checkpoint/" bucket within Amazon S3, referencing your experiment ID. This bucket will contain all of the checkpoints that were created during the experiment.

One of the main benefits of HPE Machine Learning Development Environment is its ability to automate the process of hyperparameter optimization (HPO). HPO is a process of automatically tuning the hyperparameters of a model during training, which can greatly improve a model's performance. HPE ML DE provides automated HPO, making the process of tuning and optimizing the model much easier and more efficient.

Validation loss is a metric used to measure how well the model is performing on unseen data. It is calculated by taking the difference between the predicted values and the actual values. The lower the validation loss, the better the model's performance on new data.

One key target vertical for HPE Machine Learning Development solutions is Manufacturing. Manufacturing businesses are using machine learning to automate processes, reduce costs, and improve safety and quality control. HPE ML solutions provide the tools and technologies to help manufacturers develop and deploy ML models in their production environments, enabling them to optimize and automate their operations.

If a GPU fails during a trial running on a resource pool on HPE Machine Learning Development Environment, the conductor will reschedule the trial on another available GPU in the pool, and the trial will restart from the latest checkpoint. The trial will not fail, and the ML engineer will not have to manually restart it from the latest checkpoint using the WebUI.

Adaptive ASHA is an enhanced version of ASHA that uses a reinforcement learning approach to select hyperparameter configurations. This allows Adaptive ASHA to select higher-performing configs and clone those configurations, allowing for better performance than ASHA.

The best recommendation for an ML engineer running experiments on HPE Machine Learning Development Environment to keep more of the checkpoints is to adjust the experiment config's checkpoint storage settings to save more of the latest and best checkpoints. This can be done by monitoring ongoing trials in the WebUI and clicking checkpoint flags to auto-save the desired checkpoints. Additionally, the engineer should double-check that the checkpoint storage location is operating under 90% of total capacity to ensure that enough capacity is available to store the checkpoints. Finally, they can adjust the checkpoint storage settings to save checkpoints to a shared file system instead of cloud storage if desired.