Any way to swap Enter and Shift-Enter input commands in Edit-Mode, Jupyter Notebook?

I just started using ipython/jupyter notebook. The Shift-Enter (run current cell) and Enter (insert newline) commands are frustrating to use. I would like to swap the commands for those two inputs in edit-mode.

So:

Shift-Enter: (insert newline)

Enter: (run current cell)

Is there some way to remap commands for jupyter notebook? A config file maybe? It sounds like ipython notebook did not always work this way (Enter in the IPython console inserts new line instead of executing current line after kernel restart #2696). The solution to the linked github issue seems to be "just use shift-enter," and I was unable to find a solution on google.

I have the following versions:

ipykernel (4.5.2) ipython (5.3.0) jupyter (1.0.0) notebook (4.4.1) 

1 Answers

Answers 1

Open up a notebook, and under [Help] you can find [Edit Keyboard Shortcuts]. For versions before 5.0, the documentation I linked below has a detailed explanation as to what command you can run to change the shortcuts.

Source: https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Custom%20Keyboard%20Shortcuts.html

Read More

Can't instantiate Spark Context in iPython

I'm trying to set up a stand alone instance of spark locally on a mac and use the Python 3 API. To do this I've done the following, 1. I've downloaded and installed Scala and Spark. 2. I've set up the following environment variables,

#Scala export SCALA_HOME=$HOME/scala/scala-2.12.4 export PATH=$PATH:$SCALA_HOME/bin  #Spark export SPARK_HOME=$HOME/spark/spark-2.2.1-bin-hadoop2.7 export PATH=$PATH:$SPARK_HOME/bin  #Jupyter Python export PYSPARK_PYTHON=python3 export PYSPARK_DRIVER_PYTHON=ipython3 export PYSPARK_DRIVER_PYTHON_OPTS="notebook"  #Python alias python="python3" alias pip="pip3"  export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.10.4-src.zip:$PYTHONPATH 

Now when I run the command

pyspark --master local[2] 

And type sc on the notebook, I get the following,

SparkContext  Spark UI  Version v2.2.1 Master local[2] AppName PySparkShell 

Clearly my SparkContext is not initialized. I'm expecting to see an initialized SparkContext object. What am I doing wrong here?

2 Answers

Answers 1

Well, as I have argued elsewhere, setting PYSPARK_DRIVER_PYTHON to jupyter (or ipython) is a really bad and plain wrong practice, which can lead to unforeseen outcomes downstream, such as when you try to use spark-submit with the above settings...

There is one and only one proper way to customize a Jupyter notebook in order to work with other languages (PySpark here), and this is the use of Jupyter kernels.

The first thing to do is run a jupyter kernelspec list command, to get the list of any already available kernels in your machine; here is the result in my case (Ubuntu):

$ jupyter kernelspec list Available kernels:   python2       /usr/lib/python2.7/site-packages/ipykernel/resources   caffe         /usr/local/share/jupyter/kernels/caffe   ir            /usr/local/share/jupyter/kernels/ir   pyspark       /usr/local/share/jupyter/kernels/pyspark   pyspark2      /usr/local/share/jupyter/kernels/pyspark2   tensorflow    /usr/local/share/jupyter/kernels/tensorflow 

The first kernel, python2, is the "default" one coming with IPython (there is a great chance of this being the only one present in your system); as for the rest, I have 2 more Python kernels (caffe & tensorflow), an R one (ir), and two PySpark kernels for use with Spark 1.6 and Spark 2.0 respectively.

The entries of the list above are directories, and each one contains one single file, named kernel.json. Let's see the contents of this file for my pyspark2 kernel:

{  "display_name": "PySpark (Spark 2.0)",  "language": "python",  "argv": [   "/opt/intel/intelpython27/bin/python2",   "-m",   "ipykernel",   "-f",   "{connection_file}"  ],  "env": {   "SPARK_HOME": "/home/ctsats/spark-2.0.0-bin-hadoop2.6",   "PYTHONPATH": "/home/ctsats/spark-2.0.0-bin-hadoop2.6/python:/home/ctsats/spark-2.0.0-bin-hadoop2.6/python/lib/py4j-0.10.1-src.zip",   "PYTHONSTARTUP": "/home/ctsats/spark-2.0.0-bin-hadoop2.6/python/pyspark/shell.py",   "PYSPARK_PYTHON": "/opt/intel/intelpython27/bin/python2"  } } 

Now, the easiest way for you would be to manually do the necessary changes (paths only) to my above shown kernel and save it in a new subfolder of the .../jupyter/kernels directory (that way, it should be visible if you run again a jupyter kernelspec list command). And if you think this approach is also a hack, well, I would agree with you, but it is the one recommended in the Jupyter documentation (page 12):

However, there isn’t a great way to modify the kernelspecs. One approach uses jupyter kernelspec list to find the kernel.json file and then modifies it, e.g. kernels/python3/kernel.json, by hand.

If you don't have already a .../jupyter/kernels folder, you can still install a new kernel using jupyter kernelspec install - haven't tried it, but have a look at this SO answer.

If you want to pass command-line arguments to PySpark, you should add the PYSPARK_SUBMIT_ARGS setting under env; for example, here is the last line of my respective kernel file for Spark 1.6.0, where we still had to use the external spark-csv package for reading CSV files:

"PYSPARK_SUBMIT_ARGS": "--master local --packages com.databricks:spark-csv_2.10:1.4.0 pyspark-shell" 

Finally, don't forget to remove all the PySpark/Jupyter-related environment variables from your bash profile (leaving only SPARK_HOME and PYSPARK_PYTHON should be OK).

Another possibility could be to use Apache Toree, but I haven't tried it myself yet.

Answers 2

Documentation seams to say that environment variables are read from a certain file and not as shell environment variables.

Certain Spark settings can be configured through environment variables, which are read from the conf/spark-env.sh script in the directory where Spark is installed

Read More

How to Copy from IPython session without terminal prompts

Frequently, my workflow involves data cleaning/munging in an IPython shell. This has become particularly wonderful since IPython version 5.0 with all the great upgrades to the terminal interface. So, let's say I make an attempt at sprucing up some piece of unstructured data:

In [11]: for i, (num, header, txt) in enumerate(data):     ...:     header = [e.strip() for e in header.strip().split('\n')]     ...:     header[4] = header[4].strip(',').split(',')     ...:     data[i] = (num, header, txt)     ...: 

Fantastic, it works! But now, I would really like to add this to a script in my editor. If I copy and paste from my terminal, I capture all the junk on the left. I can clean this up more-or-less easily in an editor, but it would be great if I could copy the code directly to my clipboard from the terminal without touching the mouse and without grabbing the extra stuff either. Is there such a functionality in IPython?

5 Answers

Answers 1

You can use the %history magic to extract the interesting parts from your session. They will be shown in terminal without any of the junk.

Example

In [1]: import numpy as np     In [2]: a = np.random(10) --------------------------------------------------------------------------- TypeError                                 Traceback (most recent call last) <ipython-input-2-83ce219ad17b> in <module>() ----> 1 a = np.random(10)  TypeError: 'module' object is not callable  In [3]: a = np.random.random(10) In [4]: for i in a:    ...:     print(i)    ...:      0.688626523886 [...] 0.341394850998 

If I want to save a part of the session above I can use:

In [5]: %history 1 3-4  import numpy as np a = np.random.random(10) for i in a:     print(i) 

In the example above I used %history 1 3-4 to assemble all the commands I want to keep and omit the ones I do not need (Line 2, the one with the error). Now you have version of your session that can be nicely copied.

Writing a file

You can also directly write this to file using the -f FILENAME as parameter.

In [8]: %history 1 3-4 -f /tmp/foo.py 

Be careful though, this will overwrite existing files. More Details can be found in the documentation of the %history magic.

Answers 2

So, I have finally found a great solution that is essentially exactly what I wanted: Use Vi mode in IPython. On version 5, this requires:

$ ipython --TerminalInteractiveShell.editing_mode=vi 

Now I can use handy vi-like visual mode and yank whatever I need!

Which leads to the following new alias in my .bash_profile/.bash_rc:

alias vpython='ipython --TerminalInteractiveShell.editing_mode=vi' 

Answers 3

In the shell you can first convert the IPython file to a regular Python file (.py) and then do the clean up:

http://ipython.org/ipython-doc/3/notebook/nbconvert.html (see --to script format)

You can also download the file in the notebook editor as Python file and perform the cleanup after this step.

Answers 4

I don't think terminal applications really get access to the copy/paste buffer. You're going to have to use the mouse. How do do it depends on what terminal you're using. Most modern terminals have some sort of "rectangular select" or "block select" mode.

With Windows, rectangular select is the default for cmd.exe and Powershell. If you're using Cygwin's mintty, hold Alt and then select the region with the mouse. The same goes for PuTTY.

On Linux (which I don't have in front of me - take these with a grain of salt), xterm doesn't support it, Gnome Terminal uses Ctrl as the modifier, and KDE's Konsole uses Ctrl+Alt.

For OS X Terminal, the Internet tells me that you use ⌘ while clicking.

Other terminals (and GNU Screen) likely have the feature, it's just a matter of figuring out how to activate it.

Answers 5

The save magic command [documentation] saves the input lines you want to a file; the -a option is for "append" mode so that the lines are added at the end of the file instead of overwriting the file. I use it all the time.

With you example:

%save -a myfile.py 11 # the '%' is not necessary save -a myfile.py 11 

Then you can keep coding in IPython.

When there is another command you want to write to the same file, you can just type save then use the up arrow to bring back the last use of "save" (so that the -a option and the filename are already there) and just edit the line number. Note that you can give several lines to save and also line ranges:

save -a myfile.py 15 18 19-25 

Read More

How to Copy from IPython session without terminal prompts

Frequently, my workflow involves data cleaning/munging in an IPython shell. This has become particularly wonderful since IPython version 5.0 with all the great upgrades to the terminal interface. So, let's say I make an attempt at sprucing up some piece of unstructured data:

In [11]: for i, (num, header, txt) in enumerate(data):     ...:     header = [e.strip() for e in header.strip().split('\n')]     ...:     header[4] = header[4].strip(',').split(',')     ...:     data[i] = (num, header, txt)     ...: 

Fantastic, it works! But now, I would really like to add this to a script in my editor. If I copy and paste from my terminal, I capture all the junk on the left. I can clean this up more-or-less easily in an editor, but it would be great if I could copy the code directly to my clipboard from the terminal without touching the mouse and without grabbing the extra stuff either. Is there such a functionality in IPython?

4 Answers

Answers 1

You can use the %history magic to extract the interesting parts from your session. They will be shown in terminal without any of the junk.

Example

In [1]: import numpy as np     In [2]: a = np.random(10) --------------------------------------------------------------------------- TypeError                                 Traceback (most recent call last) <ipython-input-2-83ce219ad17b> in <module>() ----> 1 a = np.random(10)  TypeError: 'module' object is not callable  In [3]: a = np.random.random(10) In [4]: for i in a:    ...:     print(i)    ...:      0.688626523886 [...] 0.341394850998 

If I want to save a part of the session above I can use:

In [5]: %history 1 3-4  import numpy as np a = np.random.random(10) for i in a:     print(i) 

In the example above I used %history 1 3-4 to assemble all the commands I want to keep and omit the ones I do not need (Line 2, the one with the error). Now you have version of your session that can be nicely copied.

Writing a file

You can also directly write this to file using the -f FILENAME as parameter.

In [8]: %history 1 3-4 -f /tmp/foo.py 

Be careful though, this will overwrite existing files. More Details can be found in the documentation of the %history magic.

Answers 2

So, I have finally found a great solution that is essentially exactly what I wanted: Use Vi mode in IPython. On version 5, this requires:

$ ipython --TerminalInteractiveShell.editing_mode=vi 

Now I can use handy vi-like visual mode and yank whatever I need!

Which leads to the following new alias in my .bash_profile/.bash_rc:

alias vpython='ipython --TerminalInteractiveShell.editing_mode=vi' 

Answers 3

In the shell you can first convert the IPython file to a regular Python file (.py) and then do the clean up:

http://ipython.org/ipython-doc/3/notebook/nbconvert.html (see --to script format)

You can also download the file in the notebook editor as Python file and perform the cleanup after this step.

Answers 4

I don't think terminal applications really get access to the copy/paste buffer. You're going to have to use the mouse. How do do it depends on what terminal you're using. Most modern terminals have some sort of "rectangular select" or "block select" mode.

With Windows, rectangular select is the default for cmd.exe and Powershell. If you're using Cygwin's mintty, hold Alt and then select the region with the mouse. The same goes for PuTTY.

On Linux (which I don't have in front of me - take these with a grain of salt), xterm doesn't support it, Gnome Terminal uses Ctrl as the modifier, and KDE's Konsole uses Ctrl+Alt.

For OS X Terminal, the Internet tells me that you use ⌘ while clicking.

Other terminals (and GNU Screen) likely have the feature, it's just a matter of figuring out how to activate it.

Read More

Running Python startup code after modules are loaded

I'm working with Jupyter notebooks and Python kernels with a SparkContext. A coworker has written some Python code that wires Spark events with ipykernel events. When we import his module from a notebook cell, it works in all combinations we need to support: Python 2.7 and 3.5, Spark 1.6 and 2.x, Linux only.

Now we want to enable that code automatically for all Python kernels. I put the import into our sitecustomize.py. That works fine for Spark 2.x, but not for Spark 1.6. Kernels with Spark 1.6 don't get an sc anymore, and something is so screwed up that unrelated imports like matplotlib.cbook fail. When I delay that import for a few seconds using a timer, it works. Apparently, the code in sitecustomize.py is executed too early for importing the module which connects Spark with the ipykernel.

I'm looking for a way to delay that import until Spark and/or ipykernel are fully initialized. But it should still execute as part of the kernel startup, before any notebook cells get executed. I found this trick to delay code execution until sys.argv is initialized. But I don't think it can work on global variables like sc, considering that Python globals are still local to modules. So far, the best I can come up with is using a timer to check every second whether certain modules are present in sys.modules. But that isn't very reliable, because I don't know how to distinguish a module that's fully initialized from one that's still in the process of being loaded.

Any ideas on how to hook in startup code that executes late during startup? A solution that is specific to pyspark and/or ipykernel would satisfy my needs.

1 Answers

Answers 1

Hmmm, you don't really give many details about what errors you encounter.

I think the canonical way to customize startup behaviour for the ipython kernel is to setup a config file and set the exec_lines option.

For example you would put in ~/.ipython/profile_default/ipython_config.py

# sample ipython_config.py c = get_config()  c.InteractiveShellApp.exec_lines = [     'import numpy',     'import scipy' ] c.InteractiveShellApp.exec_files = [     'mycode.py',     'fancy.ipy' ] 

Read More

Outputing HTML Ipython object to disk

How can I output an IPython HTML object of the form <IPython.core.display.HTML object> to a HTML file or PDF file on disk?

1 Answers

Answers 1

You can do something like this:

from IPython.core.display import HTML  a = HTML('<a href="http://example.com">LINK TO THE WEB PAGE</a>') html = a.data with open('html_file.html', 'w') as f:     f.write(html) 

Read More

Hook the global name lookup in a python interpreter

Here is the thing, I have a proxy holding the reference to a remote module, and I put some of these proxies to the sys.modules such that I can use it just like local modules. But some other objects are put in the __builtin__ module at the remote environment (like a magic variable for convenience of debugging or referencing). I don't want to reference these vars like conn.__builtin__.var, and I have to either replace the local __builtin__ (which seems not working for replace sys.modules['__builtin__'] or to hook the global name finding rules. How? For a module you can just overload a getattr to do this. But in a interactive interpreter like IPython, who is the main module or how to do this?

I'd like to turn the local environment completely to be the remote one (for a debugging console)

UPDATE

For now I just iterate all the __builtin__.__dict__ and if there is a name that isn't in the local __builtin__. I add the name to local's __builtin__. But it's not so dynamic compare to a name lookup rule say if I can't find the name in local __builtin__ try the remote one.

here is a similar discussion.

And this question gives a simulation of module by replace it with a object in sys.modules. But this won't work for __builtin__ name lookup, I've also tried to replace the __builtin__.__getattribute__ with a custom one that will first use the original lookup followed by a custom one when failed. But global name lookup of __builtin__ never called into the __builtin__.__getattribute__ even __builtin__.__getattribute__('name') returns the desired value, __builtin__.name or name never returns one.

0 Answers

Read More

Jupyter Notebook Set Default Folder to Root

I am using Jupyter Notebook on Windows 7, and I want to set the default foler to D:. Currently, I have the following line in my jupyter_notebook_config.py:

c.NotebookApp.notebook_dir = 'D:/' 

When I open Jupyter Notebook, in the browser I receive the following message:

404 : Not Found You are requesting a page that does not exist! 

In the prompt, I get the following output:

[W 14:12:45.477 NotebookApp] ipywidgets package not installed.  Widgets are unavailable. [I 14:12:45.497 NotebookApp] Serving notebooks from local directory: D:/ [I 14:12:45.497 NotebookApp] 0 active kernels [I 14:12:45.497 NotebookApp] The IPython Notebook is running at: http://localhost:8888/ [I 14:12:45.497 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [I 14:12:45.747 NotebookApp] Refusing to serve hidden directory, via 404 Error [W 14:12:45.790 NotebookApp] 404 GET /tree (::1) 44.00ms referer=None 

But, if I change my config file to point to a folder, eveything works fine. For example, the following line in config works:

c.NotebookApp.notebook_dir = 'D:/Dropbox' 

Is there any way that I can set the Jupyter default folder to the root drive?

1 Answers

Answers 1

Refusing to serve hidden directory, via 404 Error points to no write permissions on the drive.

IF you change security permissions on your D:\, you can use it as a default folder for Jupyter Notebook. You have to turn off UAC (User Account Control settings) from the Windows Control Panel (it blocks programs from writing to the root directory for security, must login as Admin to turn it off). You'll have to run the program as Administrator. This guide here is probably the best way to do it: http://superuser.com/a/753068

Remember the UAC is there to prevent unauthorized apps from writing to your root directory, so probably not the best thing to turn off. You could alternatively map a directory as another drive letter if you're just doing the D:\ for convenience.

So in summary your error message is due to selecting a directory where Windows tries to protect you from viruses, and is locked out by apps unless you turn off those protections.

Read More

Default template for iPython notebook (using Jupyter)

In the first cell of every iPython (Jupyter) notebook, I almost always type:

%matplotlib inline import matplotlib.pyplot as plt import numpy as np 

Is there a way to make it so that this cell appears at the top of each new notebook I create by default?

For example, could I save a template .ipynb file somewhere, which is copied by iPython when creating a new notebook?

(I found this question, but it seems to be more about css than default content in cells.)

1 Answers

Answers 1

I know it may not be what you're looking for (this example is not good for working on notebooks that need to be run in multiple environments, e.g. shared), but I put the following in a file called ipython_config.py in my .ipython folder.

c.InteractiveShellApp.exec_lines = ['%matplotlib inline',     'import numpy as np',     'import scipy.constants as c',     'import scipy.integrate as sci',     'from mpl_toolkits.mplot3d import Axes3D',     'import scipy.optimize as sco' ] 

This runs before anything runs in any interactive console, including the jupyter notebook. If you want explicit boilerplating, I think that you will be disappointed (unless you want to build in the functionality for us ☺)

Read More