Work with Jupyter Notebooks
Now we have seen how to execute Python files in the compute servers, but we know there are other ways to work with Python.
Jupyter Notebook allows you to create documents and run code live so that you can have instant feedback on the output generated.
Install needed libraries
The first step to using this powerful tool is to access your virtual environment. You can use either Conda or Virtualenv. If you have already done these first steps, jump to number 4.
For Conda:
1. Access your virtual environment.
2. Install two necessary packages: ipython and ipykernel (only the first time). These packages help create a temporary kernel inside the compute servers.
For Virtualenv:
1. Access your virtual environment.
2. Install three necessary packages: jupyter, ipython and ipykernel (only the first time). These packages help create a temporary kernel inside the compute servers.
Open Jupyter Notebook
Once you have accessed your virtual environment and installed the required libraries, the procedure to open Jupyter Notebook is the same for both Conda and Virtualenv:
3. Create a virtual kernel with the libraries installed inside the virtual environment (projectname) that the Jupyter's Python will use.
4. Open Jupyter Notebook using srun inside a compute server
The previous steps are permanent, so execute this command directly if you have already done the previous steps in the past.
This will let you use it for 3 hours. When you are finished, close it as explained in the next section. Otherwise, it will close after the time set (you can run the same command to open it again). Don't worry if you forgot to save changes, because it does it periodically every 2 minutes.
This time limit is to help free the resources of the computing nodes once the work is finished. It is very important to copy the full command with all the parameters.
The following example is for the sunset partition (a subset of computing nodes), please use your enabled partition (gpi.develop, veu, grcm, spcom...). You can check it with the "sinfo" command. In addition, this example gives the process the availability of 1 GPU, 4GB of RAM and a CPU core.
On
If everything is done correctly, a lot of code will appear and the way to access the Jupyter Notebook is a local browser (yes, a browser on your laptop/PC). In this example, the Jupyter Notebook is available at sunsetc01. So you could use the URL on your local browser:
http://localhost:8888/?token=2fac938688db2fc and change localhost for the IP address of sunsetc01.
> cat /etc/hosts | grep sunsetc01
This would be the result and where you should connect from your local browser:
http://172.26.105.202:8888/?token=2fac938688db2fc
5. On the new window, click on "New" in the top-right corner to create a new file.
6. IMPORTANT: Click on "projectname" to use the Python set up in our environment.
You can now start to work with your scripts.
Next time you want to open Jupyer Notebook again, you only need to do step 4 (open Jupyter Note using srun), because the space (kernel) is already created and linked to your environment.
To clarify, this Jupyter Notebook has linked to your environment. If you want to use this tool with another environment, proceed to create a new kernel from the other workspace (step 3).
Close Jupyter Notebook
Once we are finished, we need to close the program and stop all the running processes:
1. Save your files
2. Close Jupyter Notebook (window).
3. Return to the terminal where the job is running.
4. Stop the Jupyter processes by pressing "Ctrl+C" 2 times:
Now the process is stopped. Next time you want to use Jupyter Notebook again, access your environment and repeat step 4.
(Additional) Delete the kernel based on a virtual environment not used anymore
This section is just for those who used Jupyter Notebook and are not planning on using the environment anymore. You could just leave the kernel created but it is a good idea to free space.
To do so, execute the following command (and press Y):
Share: