Jekyll2022-05-01T17:57:50+00:00http://blog.cocalc.com/feed.xmlCoCalc BlogNews and updates about CoCalcSageMath, Inc.Dash with CoCalc (2021 version)2021-05-03T00:00:00+00:002021-05-03T00:00:00+00:00http://blog.cocalc.com/cocalc/python/2021/05/03/dash-with-cocalc<p>Create interactive data visualizations for collaborators in your CoCalc projects using <a href="https://plot.ly/products/dash/">Dash</a>.</p>
<p>Dash is an open-source framework to create web applications with Python.
With CoCalc’s <a href="https://doc.cocalc.com/howto/webserver.html">Run A Webserver</a> capability, you can run a Dash application from inside a CoCalc project.</p>
<div style="text-align: center;">
<img src="/img/dash/dash-demo.png" style="width:80%" />
<br />
<em>Dash application running in a CoCalc project</em>
</div>
<p>We’ll step through a working program based on the first example in the <a href="https://dash.plot.ly/getting-started">Dash Tutorial</a>,
with a few additions for configuring the HTTP server.
Code for the <a href="https://cocalc.com/share/db982efa-e439-4e2d-933b-7c7011c6b21a/DASH/dash-demo.py?viewer=share">completed CoCalc example</a> is viewable on the share server.</p>
<p>The following packages used by the demo program are already installed in the CoCalc Python 3 environment:</p>
<ul>
<li>dash</li>
<li>dash_core_components</li>
<li>dash_html_components</li>
<li>plotly</li>
</ul>
<p>The sample program at the <a href="https://dash.plot.ly/getting-started">Dash Tutorial</a> is short enough that you can copy it into your clipboard and paste into a file in a project of your own. In the following, we assume your file is called <code class="highlighter-rouge">app.py</code>.</p>
<p>Comment out or delete the call to the constructor that begins <code class="highlighter-rouge">app = dash.Dash...</code> In its place insert the following three code snippets:</p>
<ol>
<li>
<p>Set the HTTP port number for your Dash web app. You can probably use the default port number of 8050, but you will need to specify the port number in the prefix regardless. Our example uses 9990.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>port = 9990
</code></pre></div> </div>
</li>
<li>
<p>Get your project ID from the environment variable <code class="highlighter-rouge">COCALC_PROJECT_ID</code>. That is the specially-formatted hex code you see in the browser URL when logged into your project, and displayed the Project Settings tab under Project Control.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import os
cocalc_project_id = os.environ['COCALC_PROJECT_ID']
</code></pre></div> </div>
</li>
<li>
<p>Configure <code class="highlighter-rouge">requests_pathname_prefix</code> as indicated in the CoCalc online guide’s <a href="https://doc.cocalc.com/howto/webserver.html">Run A Webserver</a> page and set requests_pathname_prefix in the Dash constructor:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pfx = "/{}/server/{}/".format(cocalc_project_id, port)
app = dash.Dash(requests_pathname_prefix = pfx)
app.config.requests_pathname_prefix = pfx
</code></pre></div> </div>
</li>
</ol>
<p>In the final line, display the URL for your server
and override default port number and
host in the call to <code class="highlighter-rouge">app.run_server()</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>if __name__ == '__main__':
print("browse to https://cocalc.com{}".format(pfx))
app.run_server(debug=True, port=port, host='0.0.0.0')
</code></pre></div></div>
<p>You will start the program from a Linux terminal session. You could <a href="https://doc.cocalc.com/terminal.html">open a separate .term file</a>. Or you can split your editor window and open one of the frames into a terminal. See the online guide about the <a href="https://doc.cocalc.com/frame-editor.html">Frame Editor</a> for more information.</p>
<p>If the file you just edited is <code class="highlighter-rouge">app.py</code>, do this in your terminal:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python app.py
</code></pre></div></div>
<p>Copy the URL that is displayed by the program.
Open a new browser tab and paste in the URL. You will see the “Hello Dash” demo.</p>
<p>Note: Because of the way CoCalc security works, only collaborators on the given project can access the web server. Our proxy server will reject all other requests.</p>
<p>More examples are available in the <a href="https://dash.plot.ly/">Dash User Guide</a> and <a href="https://medium.com/@plotlygraphs/introducing-dash-5ecf7191b503">Introducing Dash</a>.</p>Hal SnyderCreate interactive data visualizations for collaborators in your CoCalc projects using Dash. Dash is an open-source framework to create web applications with Python. With CoCalc’s Run A Webserver capability, you can run a Dash application from inside a CoCalc project. Dash application running in a CoCalc projectShould I Resign From My Full Professor Job To Work Fulltime On Cocalc?2019-04-12T00:00:00+00:002019-04-12T00:00:00+00:00http://blog.cocalc.com/2019/04/12/should-i-resign-from-my-full-professor-job-to-work-fulltime-on-cocalc<p>Nearly 3 years ago, I gave <a href="https://wstein.org/talks/2016-06-sage-bp/">a talk</a> at a Harvard mathematics conference
announcing that “I am leaving academia to build a company”.
What I really did is go on <em>unpaid leave</em> for three years
from my tenured Full Professor position. No further extensions
of that leave is possible, so I finally have to decide
whether or not to go back to academia or resign.</p>
<h2 id="how-did-i-get-here">How did I get here?</h2>
<p>Nearly two decades ago, as a recently minted Berkeley math Ph.D., I was hired
as a non-tenure-track faculty member in the mathematics department
at Harvard. I spent five years at Harvard, then I applied for
jobs, and accepted a tenured Associate Professor position in
the mathematics department at UC San Diego. The mathematics
community was very supportive of my number theory research; I skipped
tenure track, and landed a tier-1 tenured position by the time
I was 30 years old. In 2006, I moved from UCSD to a tenured
Associate Professor position at the University of Washington (UW)
mathematics department, primarily because my wife was a graduate
student there, UW has strong research in number theory and algebraic
geometry, and they have a good
culture supporting undergraduate research.</p>
<p>Before I left Harvard, I started the <a href="https://sagemath.org">SageMath open source software project</a>, initially with
the longterm goal of creating a free open source viable alternative
to Mathematica, Maple, Matlab and Magma.
As a result, in addition to publishing <a href="https://wstein.org/papers/">dozens of
research mathematics papers</a> and
<a href="https://wstein.org/books/">some books</a>, I also started spending
a lot of my time writing software, and
organizing <a href="https://wiki.sagemath.org/Workshops#Past_Workshops">Sage Days workshops</a>.</p>
<h2 id="recruiting-at-uw-mathematics">Recruiting at UW Mathematics</h2>
<p>At UW, I recruited an
amazing team of undergraduates and grad students
who had a major impact on the development of Sage.
I was blown away by
the quality of the students (both undergrad and grad)
that I was able to get involved in Sage development.
I fully expected that in the next few years I would have the resources
to hire some of these students to work fulltime on Sage.
They had written the first versions of much of the core functionality
of Sage (e.g., graph theory, symbolic calculus, matrices, and much more).</p>
<p>I was surprised when my application for Full Professor at
UW was delayed for one year because – I was told – I wasn’t
publishing enough research papers. This was because I was working
very hard on building Sage, which was going extremely well at the time.
I took the feedback seriously, and put more time into traditional
research and publishing; this was the first time in my life that I did research
mathematics for reasons other than just because I loved doing it.</p>
<p>I tried very hard to hire <a href="http://wbhart.blogspot.com/">Bill Hart</a>
as a tenure-track faculty member at UW.
However, I was told that his publication count was <em>“a bit light”</em>, and I did not
succeed at hiring him. If you printed out the source
code of software he has written, it would be a tall stack of paper.
In any case, I totally failed at the politics needed to make
his case and was left dispirited, realizing my
personal shortcomings at department politics meant I probably
could not hire the sort of colleagues I desperately needed.</p>
<p>UW was also very supportive of me teaching
an undergrad course on open source math software (it evolved
into <a href="https://github.com/kedlaya/math157">this</a>). I taught
a similar course at the graduate level once, and it went
extremely well, and was in my mind the best course I ever
taught at UW. I was extremely surprised when my
application to teach
that grad course again was denied, and I was told that
grad students should just go to my undergraduate course.
I thought, “this is really strange”, instead of
lobbying to teach the course and better presenting
my case.</p>
<p>To be clear, I do not mean to criticize the mathematics department.
The UW math department has thought very hard and systematically about
their priorities and how they fit into UW. They are a traditional <em>pure</em> mathematics
departments that is generally ranked around 25 in the country,
with a particular set of strengths. There is a separate
applied math department on campus, several stats departments, and
a massive <strong>School</strong> of Computer Science. Maybe I was in the
wrong place to try to hire somebody whose main qualification
is being world class at writing mathematical software. This blog post is
about the question of whether the UW math department is the right
place for me or not.</p>
<h2 id="outside-grant-support">Outside Grant Support?</h2>
<p>My number theory research received incredible support from the NSF,
with me being the PI on <a href="https://wstein.org/grants/">six NSF grants</a>.
Also, <a href="https://magma.maths.usyd.edu.au">Magma</a> (which is similar to Sage, but
closed source) had managed to find
sufficient government funding, so I remained optimistic. Maybe I could
fund people to build Sage via grants, and even start an institute!
I applied for grants to support work on SageMath at a larger scale,
and had some initial success (half of a postdoc, and
some workshops, etc.).</p>
<p>Why is grant funding so important for Sage?
The goal of the SageMath project is to create free open source
software that is a viable alternative to Mathematica, Maple, Matlab, and
Magma – software produced by companies with a combined thousands of fulltime employees.
Though initial progress was encouraging,
it was clear that I desperately needed significant money to
genuinely compete. For example, one Sage developer had
a fantastic Sage development project he wanted about 20K to work
fulltime on during a summer, and I could not find the money;
as a result he quit working on Sage. This project involved implementing some
deep algorithms that are needed to more directly compete with
Mathematica for solving symbolic inequalities.
This sort of thing happened over and over again,
and it began to really frustrate me. I could get plenty of funding for
1-week workshops (just travel expenses – everybody works for free),
but there’s only so much you can do at such sprints.</p>
<p>I kept hearing that there would be a big one-in-10-years
NSF institutes competition sometime in the “next year or two”.
People hinted to me that this would be a good thing to watch
out for, and I dreamed that I could found such an institute, with
the mission to make it so the mathematics
community finally owned the deep software on which teaching
and research are based. This institute would bring the same
openness and robustness to computational mathematics that
rigorous proof had brought to mathematics itself a century
earlier.</p>
<p>Alas, this did not happen. I remember the moment I found
out about the actual NSF institutes competition. <a href="http://www.math.brown.edu/~jhs/">Joe
Silverman</a> was standing behind me
at a coffee break at <a href="http://swc.math.arizona.edu/aws/2010/index.html">The Arizona Winter School 2010</a> telling people
about how <a href="https://icerm.brown.edu/">his proposal for ICERM</a>
had just won the NSF institutes competition.
I spun around and congratulated him as I listened to how much work it was to put
together the application during the last year; internally,
my heart sunk. Not only did I not win, I didn’t even know the
competition had happened! I guess I was too busy working on Sage.
In any case, my fantasy of creating an NSF-funded institute
died at that moment. Of course, ICERM has turned out to be a fantastic
institute, and it has hosted several workshops that support
the development of open source math software.</p>
<p>Around this time, I also started <a href="https://wstein.org/grants/">having my grant proposals denied</a> for reasons I do not understand.
This was confusing to me, after having received so many
NSF grants before.
In 2012, the Simons Foundation put out a call for something
that potentially addressed what I had hoped to accomplish via
an NSF-funded institute. I was very excited again, but
that <a href="http://sagemath.blogspot.com/2015/09/the-simons-foundation-and-open-source.html">did not turn out as I had hoped</a>. So next I tried something I never
thought I would ever do in a million years…</p>
<h2 id="commercialization-at-uw">Commercialization at UW</h2>
<p>For various reasons, I failed to get the NSF or other foundations to
fund Sage at the level I needed, so in 2013,
I decided to try to sell
a commercial product, and use the profits to fund Sage development.
I first tried to do this at University of Washington, by working
with <a href="https://comotion.uw.edu/">the commercialization office (C4C)</a>
to sell access to Sage online. As long as the business and product were
merely abstract ideas (e.g., let’s make up a name and trademark it!
let’s write some terms of service!) things went fine. However, when things
became much more concrete, working with C4C got strange and frustrating
for me. I was clearly missing something.</p>
<p>For example, the first thing C4C told me on the very first day we
sat down together was they would not work with me if I made the software
I wrote for this open source, and that the university would own the software.
Given there was no software at all yet, and I imagined I would just whip
out a quick modern web-based frontend to Sage and make boatloads of money
that would go straight into a UW account to be
used to fund Sage, this seemed fine to me. However, I had a nagging feeling
that a pure closed-source approach to this problem was impossible, and not
having that flexibility would come back to haunt me.</p>
<p>Naively optimistic, I found myself working fulltime at UW and
at the same time trying to get a
sophisticated web application off the ground by myself, with many
important early users
depending on it for their classes. This was stressful and took an enormous
amount of time and attention. I felt like I was just part of the software, often getting
warnings that things were broken or breaking, and <strong>manually</strong> fixing them.
The toil was high, and only got worse as more people used the software.
I would get woken up all night.
I couldn’t travel since things were constantly breaking.</p>
<blockquote>
<p><em>Every time I fought through
some really difficult problem with the web application
instead of just giving up, I came
out far more determined not to quit.</em></p>
</blockquote>
<p>The web application described above evolved over 6 years into what
is now https://CoCalc.com; the functionality was pretty similar from
day 1, but quality and scalability have come a long ways. CoCalc
lets you collaboratively use LaTeX, Sage, Terminals, Jupyter Notebooks,
etc., for teaching and research.</p>
<p>In 2014, I went on sabbatical and worked fulltime developing this
web application and the feedback loop I described above only grew
more intense: fix things, fight through difficult problems,
be even more determined not to give up.
Fortunately, I had some leftover NSF grant funds, and was able to
use them to hire several students to help with
development. I failed to find students
who I could hire to do the backend work (and be available any time
day or night),
which meant that much of the stress
of keeping the site running continued to fall squarely on my shoulders.
And as the site grew in popularity (and functionality), the stress
from it got worse.</p>
<p>My Sabbatical ended, and I was required
to return to UW fulltime for one year,
or return all the money I was paid during my sabbatical.
So far, CoCalc had grown in popularity, but I had not been allowed
by the “commercialization office” to actually commercialize it, so it
was still a free site.</p>
<p>I taught at UW at the same time as being the main
person trying to run this very complicated and painful production web application.
Based on user feedback,
I was also highly motivated to improve CoCalc. I would typically
sleep a few hours, get up at 3am and write code until 8am, then prepare
to teach, hope not to have any site issues right before class, and
so on. One day CoCalc got hit by a massive DDoS attack minutes before
a class I was teaching, while I was talking with a prospective donor
to the math department.</p>
<p>I am the sort of person who does well focusing on exactly one thing
at a time. Given the chance to fully focus on one thing for extended
periods of time, I sometimes even do things that really matter and
have an impact. I am not great at doing many different things at once.</p>
<p>In the meantime, Sage itself was growing and receiving funding, though this
had nothing to do with me.
For example, Gregg Musiker was putting together a <a href="https://www.ima.umn.edu/2017-2018.2/W8.21-25.17">big program at
IMA</a>, in the form
of a ton of Sage Days workshops. Also, the
<a href="https://opendreamkit.org/">huge ODK project</a>, which was a European
Union grant proposal to support open source math software would be fully funded.
And closer to home, Moore and Sloane funded a <a href="https://escience.washington.edu/">major new initiative</a> that could potentially have also
supported work on Sage.
I was invited to go to workshops and events involving these and other
grants, but often I either said no or canceled at the last minute
due to the toil needed just to keep CoCalc running. Also, I believed
if I could start charging customers, then I would have a lot more
money, and could hire more help.</p>
<p>I met with more senior people at UW’s C4C to
finally actually charge people to use CoCalc. They wanted me
to do some integration with their license management system, and sell
“express” software licenses. It didn’t make any sense to me,
and we went around in circles. I then asked about
actually starting a separate company (a legal entity) that the university
would have some ownership in, so that the company could take payments,
etc. This is when things got really weird. They would not talk with
me about creating the company due to “conflict of interest”.</p>
<p>I searched for other UW faculty that had commercialized remotely
similar products, and found one. He told me how it went, and
said it was the worst experience of his life. UW owned
50% of the company, and all of the software of the
company, which they licensed under onerous terms. They refused
to negotiate anything with him, instead requiring his spinoff company to
hire an outside negotiator. As a result of all this,
I educated myself as much as possible about relevant
rules and laws, and consulted with a lawyer.</p>
<p>It turns out that the NSF grants I used to fund work on CoCalc
explicitly stipulated that code funded by those grants had
to be GPL licensed. <a href="https://news.ycombinator.com/item?id=8967094">This meant all the code for CoCalc
had to be open sourced.</a>
Later the university even agreed
in writing to release a snapshot of all the CoCalc code
under the BSD license, and I haven’t been paid a penny by
UW since the date of that release, so there is no possible
claim that the company can’t use the code.</p>
<h2 id="building-a-company">Building a company</h2>
<p>A colleague of mine from when I was at
Harvard was in town for a day, and we met for coffee. He expected
we would talk about Sage and number theory, but
instead I told him about CoCalc and my attempts at commercialization
and starting a company. He immediately suggested a solution to
my problems, which was to talk with a friend of his who had both
extensive experience working with companies and deep connections with
mathematics. I was confident that in the
worst case I could quit my job at UW and rewrite all the software
from scratch, so I took him up on the offer.</p>
<p>In 2015 I formed a corporation, and received some outside investment,
and used that (and dramatically cutting my already-small academic
income) to “leave academia”. More precisely, in 2016 (after working fulltime
for a year at UW), I finally went
on 100% unpaid leave from UW in order to
completely focus on CoCalc development and getting a business off the ground.
Also, there was no good reason
to quit a tenured Full Professor job when you can go
on leave; also CoCalc supports teaching in math departments,
so it is closely related to my academic job.
The only academic responsibilities
I had were to my two Ph.D. students, who I meet with one-on-one
at least once a week. At the end of two years, I requested
a third year of unpaid leave, which UW granted (this is not routine).
Throughout all this, the UW mathematics department was very supportive.</p>
<p>During these three years on unpaid leave, I’ve hired <a href="http://blog.sagemath.com/cocalc/2018/09/10/where-is-cocalc-from.html">three other
people</a> who work fulltime on CoCalc. Together we have massively improved
the software, and built a business with thousands of paying
customers. The company is still not profitable,
though the future is clearly very bright if we continue what we
are currently doing. CoCalc has become a platform that
an increasing number of scalable products (such as
<a href="https://wit.io/portfolio/minerva-active-learning-forum">this</a>)
are being built with, and there is enormous growth potential
in the next year.</p>
<p>At this point, <a href="https://github.com/sagemath/website/pull/162#issue-248232293">it rightfully appears to the community</a>
that I have left SageMath development to
focus fulltime on building CoCalc as an independent business.
Indeed, I do not spend any significant time contributing to Sage,
and I even switched to getting daily digests of <a href="https://groups.google.com/forum/#!forum/sage-devel">the sage-devel mailing list</a>.</p>
<p>On the other hand, as mentioned above, CoCalc is going
well by many metrics (in terms of quality, feature development,
customer love, market position, etc.).
Most importantly, me and the other three people
who work fulltime on CoCalc really, really love this job,
and the potential to have a significant impact.
I still don’t know if CoCalc will ever be wildly profitable and
massively fund Sage
development. If I were to
obsess over only that goal, I would have to quit
working on CoCalc (since it is taking way too long) and
pursue other opportunities for funding Sage.</p>
<blockquote>
<p><em>In retrospect, my idea from 7 years ago to start a web-based
software company from scratch and build it into a successful
profitable business has so far completely failed to fund Sage.</em></p>
</blockquote>
<p>It would be far easier to work fulltime writing grants to foundations,
and leveraging the <a href="http://www.sigsam.org/awards/jenks/awardees/2013/">acknowledged success</a>
of Sage so far. I made the wrong move, given my original goal. The
surprise is that <strong>I really enjoy</strong> what I’m doing right now!</p>
<h2 id="my-unpaid-leave-is-up--what-am-i-going-to-do">My unpaid leave is up – what am I going to do?</h2>
<p>My third year of unpaid leave from UW is up. I have to decide whether
to return to UW or resign. If I return, it turns out that I
would have to have <a href="https://ap.washington.edu/ahr/academic-titles-ranks/professor/">at least a 50% appointment</a>.
I currently have 50% of one year of teaching in “credits”, which means I wouldn’t be required
to teach for the first year I go back as a 50% appointment.
Moreover, the current department chair (John Palmieri) understands and
appreciates Sage – he is among the <a href="https://github.com/sagemath/sage/graphs/contributors">top 10 all time contributors
to the source code of Sage</a>!</p>
<p>I have decided to resign. I’m worried about issues of intellectual
property; it would be extremely unfair to my employees, investors and customers
if I took a 50% UW position, and then later got sued by UW as
a result. Having a 50% paid appointment at UW subjects one to a
lot of legal jeopardy, which is precisely why I have been on 100% unpaid
leave for the last three years.
But more importantly, I feel very good about continuing
to focus 100% on the development of CoCalc, which is going
to have an incredible year going forward. I genuinely love
building this (non-VC funded) company, and feel very good about it.</p>
<hr />
<p><a href="https://news.ycombinator.com/item?id=19682126">Hacker News Discussion</a></p>William SteinNearly 3 years ago, I gave a talk at a Harvard mathematics conference announcing that “I am leaving academia to build a company”. What I really did is go on unpaid leave for three years from my tenured Full Professor position. No further extensions of that leave is possible, so I finally have to decide whether or not to go back to academia or resign. How did I get here? Nearly two decades ago, as a recently minted Berkeley math Ph.D., I was hired as a non-tenure-track faculty member in the mathematics department at Harvard. I spent five years at Harvard, then I applied for jobs, and accepted a tenured Associate Professor position in the mathematics department at UC San Diego. The mathematics community was very supportive of my number theory research; I skipped tenure track, and landed a tier-1 tenured position by the time I was 30 years old. In 2006, I moved from UCSD to a tenured Associate Professor position at the University of Washington (UW) mathematics department, primarily because my wife was a graduate student there, UW has strong research in number theory and algebraic geometry, and they have a good culture supporting undergraduate research. Before I left Harvard, I started the SageMath open source software project, initially with the longterm goal of creating a free open source viable alternative to Mathematica, Maple, Matlab and Magma. As a result, in addition to publishing dozens of research mathematics papers and some books, I also started spending a lot of my time writing software, and organizing Sage Days workshops. Recruiting at UW Mathematics At UW, I recruited an amazing team of undergraduates and grad students who had a major impact on the development of Sage. I was blown away by the quality of the students (both undergrad and grad) that I was able to get involved in Sage development. I fully expected that in the next few years I would have the resources to hire some of these students to work fulltime on Sage. They had written the first versions of much of the core functionality of Sage (e.g., graph theory, symbolic calculus, matrices, and much more). I was surprised when my application for Full Professor at UW was delayed for one year because – I was told – I wasn’t publishing enough research papers. This was because I was working very hard on building Sage, which was going extremely well at the time. I took the feedback seriously, and put more time into traditional research and publishing; this was the first time in my life that I did research mathematics for reasons other than just because I loved doing it. I tried very hard to hire Bill Hart as a tenure-track faculty member at UW. However, I was told that his publication count was “a bit light”, and I did not succeed at hiring him. If you printed out the source code of software he has written, it would be a tall stack of paper. In any case, I totally failed at the politics needed to make his case and was left dispirited, realizing my personal shortcomings at department politics meant I probably could not hire the sort of colleagues I desperately needed. UW was also very supportive of me teaching an undergrad course on open source math software (it evolved into this). I taught a similar course at the graduate level once, and it went extremely well, and was in my mind the best course I ever taught at UW. I was extremely surprised when my application to teach that grad course again was denied, and I was told that grad students should just go to my undergraduate course. I thought, “this is really strange”, instead of lobbying to teach the course and better presenting my case. To be clear, I do not mean to criticize the mathematics department. The UW math department has thought very hard and systematically about their priorities and how they fit into UW. They are a traditional pure mathematics departments that is generally ranked around 25 in the country, with a particular set of strengths. There is a separate applied math department on campus, several stats departments, and a massive School of Computer Science. Maybe I was in the wrong place to try to hire somebody whose main qualification is being world class at writing mathematical software. This blog post is about the question of whether the UW math department is the right place for me or not. Outside Grant Support? My number theory research received incredible support from the NSF, with me being the PI on six NSF grants. Also, Magma (which is similar to Sage, but closed source) had managed to find sufficient government funding, so I remained optimistic. Maybe I could fund people to build Sage via grants, and even start an institute! I applied for grants to support work on SageMath at a larger scale, and had some initial success (half of a postdoc, and some workshops, etc.). Why is grant funding so important for Sage? The goal of the SageMath project is to create free open source software that is a viable alternative to Mathematica, Maple, Matlab, and Magma – software produced by companies with a combined thousands of fulltime employees. Though initial progress was encouraging, it was clear that I desperately needed significant money to genuinely compete. For example, one Sage developer had a fantastic Sage development project he wanted about 20K to work fulltime on during a summer, and I could not find the money; as a result he quit working on Sage. This project involved implementing some deep algorithms that are needed to more directly compete with Mathematica for solving symbolic inequalities. This sort of thing happened over and over again, and it began to really frustrate me. I could get plenty of funding for 1-week workshops (just travel expenses – everybody works for free), but there’s only so much you can do at such sprints. I kept hearing that there would be a big one-in-10-years NSF institutes competition sometime in the “next year or two”. People hinted to me that this would be a good thing to watch out for, and I dreamed that I could found such an institute, with the mission to make it so the mathematics community finally owned the deep software on which teaching and research are based. This institute would bring the same openness and robustness to computational mathematics that rigorous proof had brought to mathematics itself a century earlier. Alas, this did not happen. I remember the moment I found out about the actual NSF institutes competition. Joe Silverman was standing behind me at a coffee break at The Arizona Winter School 2010 telling people about how his proposal for ICERM had just won the NSF institutes competition. I spun around and congratulated him as I listened to how much work it was to put together the application during the last year; internally, my heart sunk. Not only did I not win, I didn’t even know the competition had happened! I guess I was too busy working on Sage. In any case, my fantasy of creating an NSF-funded institute died at that moment. Of course, ICERM has turned out to be a fantastic institute, and it has hosted several workshops that support the development of open source math software. Around this time, I also started having my grant proposals denied for reasons I do not understand. This was confusing to me, after having received so many NSF grants before. In 2012, the Simons Foundation put out a call for something that potentially addressed what I had hoped to accomplish via an NSF-funded institute. I was very excited again, but that did not turn out as I had hoped. So next I tried something I never thought I would ever do in a million years…Running Your own Free CoCalc Docker Server on Google Cloud Platform2018-11-07T00:00:00+00:002018-11-07T00:00:00+00:00http://blog.cocalc.com/cocalc/2018/11/07/cocalc-docker-gcp<h2 id="introduction">Introduction</h2>
<p><a href="https://cocalc.com">CoCalc</a> is a web application that lets you collaboratively
use a large amount of free open source
math and data related open source software. You can create
collaborative <a href="http://jupyter.org/">Jupyter notebooks</a>, edit LaTeX documents, use
Terminals, use <a href="http://blog.sagemath.com/cocalc/2018/11/05/x11.html">graphical Linux applications</a>, create chatrooms,
and much more. There’s extensive support for <a href="http://www.sagemath.org/">SageMath</a>, <a href="https://www.gnu.org/software/octave/">Octave</a> (a MATLAB clone),
and <a href="https://www.r-project.org/">R</a>. See <a href="https://doc.cocalc.com/">the docs</a>.</p>
<p><a href="https://github.com/sagemathinc/cocalc-docker">Cocalc-docker</a>
is a completely free and open source self contained version of
CoCalc, which you can run on your own computer or cloud server.
This post is about how to freely play around with running
CoCalc-docker on Google Cloud Platform.</p>
<p>If you scroll down and see all the cool things CoCalc can do, but
don’t want to bother running your own server, make an
account at <a href="https://cocalc.com">CoCalc</a> and use our hosted
service, which has filesystem snapshots, a vast amount of preinstalled software (much more than cocalc-docker), and
support.</p>
<h2 id="sign-up-for-google-cloud-platform">Sign up for Google Cloud Platform</h2>
<div style="text-align: center;">
<a href="https://cloud.google.com/free/" target="_blank"><img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.34.45 AM.png" style="width:90%" /></a>
<br />
<em>Click above to learn about Google Cloud Platform's free trial</em>
</div>
<h2 id="create-a-container-instance">Create a Container Instance</h2>
<h3 id="where-to-run-it">Where to run it?</h3>
<p>Choose a location near you:</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.31.32 AM.png" style="width:90%" />
<br />
<em>Choose where to run the container close to you, for optimal speed!</em>
</div>
<h3 id="select-to-run-a-container-directly-and-configure-your-machine-type">Select to run a container directly (and configure your machine type)</h3>
<p>Click the checkbox next to “Deploy a container image to this VM instance.”, then
put <code class="highlighter-rouge">sagemathinc/cocalc</code> in the blank below “Container image”. Also check
the boxes next to buffer stdin and allocate a tty.</p>
<p>You can also change the machine type, though the default will work.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.32.19 AM.png" style="width:90%" />
<br />
<em>Container Image and Machine type</em>
</div>
<h3 id="increase-base-image-size-to-at-least-20gb">Increase Base Image Size to at least 20GB!</h3>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.32.47 AM.png" style="width:90%" /><a href="http://www.sagemath.org/">http://www.sagemath.org/</a>
<br />
<em><b>CRITICAL:</b> increase the base image size to at least 20GB! The default of 10GB will fail.</em>
</div>
<h3 id="allow-the-container-to-provide-an-httphttps-web-server">Allow the container to provide an HTTP/HTTPS web server</h3>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.32.53 AM.png" style="width:90%" />
<br />
<em>Enable https and http access</em>
</div>
<h3 id="make-the-instance-pre-emptible-optional">Make the instance pre-emptible (optional)</h3>
<p>If you are just playing around to test this out, open “Management, security, disks, networking, sole tenancy” and
scroll down and set “Preemptibility” to On. This will make things <strong>way cheaper</strong> (using less of your free trial credits). This is especially useful if
you wan to do a relatively quick but very CPU intensive parallel computation.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.33.09 AM.png" style="width:90%" />
<br />
<em></em>
</div>
<p>Here’s how our cost estimate comes out so far, with preemptible on. It would be about <strong>four times as expensive</strong> without preemptible on.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 10.57.06 AM.png" style="width:90%" />
<br />
<em>Pretty cheap...</em>
</div>
<p>Of course the drawback of preemptible is that the machine will be killed within 24 hours. That’s fine for
testing things out though.</p>
<h3 id="click-create-at-the-bottom-to-start-creating-your-vm">Click Create at the bottom to start creating your VM</h3>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.33.43 AM.png" style="width:90%" />
<br />
<em>You'll see this line in your list of VM's when the instance is being created</em>
</div>
<h3 id="watch-the-serial-port">Watch the Serial Port</h3>
<p>Once the VM is created, click to open it, then click “Serial port 1 (console)” (or “Connect to serial console”), to
watch the log as the machine boots up.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.37.34 AM.png" style="width:90%" />
<br />
<em>Watch the Serial Port</em>
</div>
<p>It takes at least <strong>10 minutes</strong> to pull and decompress the <code class="highlighter-rouge">sagemathinc/cocalc</code> Docker image. If this fails, you
probably forgot to increase the size of the boot disk from 10GB to 20GB (or more), in which case you should delete everything
and start over.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.44.21 AM.png" style="width:90%" />
<br />
<em>Wait at least 10 minutes until you see the above</em>
</div>
<h3 id="determine-the-ip-address">Determine the IP address</h3>
<p>Once your machine is running <em>and</em> the <code class="highlighter-rouge">sagemathinc/cocalc</code> image has been pulled and decompressed,
find the external ip address of your machine, and open it in a new browser tab. In my case, I
open <code class="highlighter-rouge">https://35.227.184.91/</code>.</p>
<p>Do <strong>NOT</strong> choose the address that starts <code class="highlighter-rouge">10.</code>, since that is internal.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.44.53 AM.png" style="width:90%" />
<br />
<em>Copy your IP address</em>
</div>
<p>If this fails, you probably forgot to check the box next to “Allow HTTPS traffic”.</p>
<h3 id="security-warning">Security warning</h3>
<p>Since the SSL cert in the Docker image is self-signed, you’ll get a warning. Click through it by
clicking “ADVANCED”.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.45.00 AM.png" style="width:90%" />
<br />
<em>Click ADVANCED</em>
</div>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.45.06 AM.png" style="width:90%" />
<br />
<em>Click Proceed...</em>
</div>
<h3 id="create-a-new-account-on-your-personal-cocalc-server">Create a new account on your personal CoCalc server</h3>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.45.16 AM.png" style="width:90%" />
<br />
<em>Click to create an account...</em>
</div>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.45.37 AM.png" style="width:90%" />
<br />
<em>Create an account</em>
</div>
<p><strong>WARNING:</strong> Anybody who knows the IP address can make an account in the same way. There’s no
secret token, and currenly no way to configure one with GCP Container Image. See <a href="https://github.com/sagemathinc/cocalc-docker/issues/34">this issue</a>.</p>
<h3 id="connected">Connected?</h3>
<p>Click the network icon in the upper right:</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.57.35 AM.png" style="width:90%" />
<br />
<em>Your ping time should be small</em>
</div>
<h3 id="make-a-test-project">Make a test project</h3>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.45.45 AM.png" style="width:90%" />
<br />
<em>Make a project</em>
</div>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.46.10 AM.png" style="width:90%" />
<br />
<em>Open your project</em>
</div>
<h3 id="lets-take-our-project-for-a-spin">Let’s take our project for a spin</h3>
<p>Click “+New” (or click the big “Create or Upload” button) to show the new file page:</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.46.26 AM.png" style="width:90%" />
<br />
<em>Start using it by creating a file</em>
</div>
<p>Make whatever files you want and play around with them. I’ll make a few below, just for fun, to
give you a sense of what is possible.</p>
<h3 id="the-octave-graphical-interface">The Octave Graphical Interface</h3>
<p>Octave is a MATLAB clone, and you can run the graphical UI of Octave by clicking “X11 Desktop”,
then typing <code class="highlighter-rouge">octave</code>, and clicking the tab on the right when it appears. I copied some <a href="http://octave.org/doc/v4.4.0/Three_002dDimensional-Plots.html">code to draw a 3d plot</a>:</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.50.16 AM.png" style="width:90%" />
<br />
<em>Using the Octave Graphical Interface</em>
</div>
<p><a href="http://blog.sagemath.com/cocalc/2018/11/05/x11.html">Graphical Linux applications</a> should be pretty fast, since
the server is dedicated to you and should be very close to you.</p>
<p>In the image above, I split the X11 frame to show the graphics
on one side and the Octave UI on the other.</p>
<h3 id="a-jupyter-notebook">A Jupyter Notebook</h3>
<p>Next, let’s create a Jupyter notebook:</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.50.44 AM.png" style="width:90%" />
<br />
<em>In the +New page, click on "Jupyter notebook".</em>
</div>
<p>Once the notebook appears, I set the kernel to Octave, and pasted the same code in:</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.51.27 AM.png" style="width:90%" />
<br />
<em>Drawing a Sombrero using Jupyter and Octave</em>
</div>
<h3 id="a-sage-worksheet">A Sage Worksheet</h3>
<p>Next, let’s make a <a href="http://www.sagemath.org/">SageMath</a> worksheet; of course CoCalc-docker comes with the latest version of Sage.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.51.36 AM.png" style="width:90%" />
<br />
<em>Click the Sage worksheet button</em>
</div>
<p>Let’s draw an interactive 3d plot using Sage.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.53.01 AM.png" style="width:90%" />
<br />
<em>Choose a plot from the drop down menu</em>
</div>
<p>You should be able to rotate 3d plots around with your mouse:</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.53.18 AM.png" style="width:90%" />
<br />
<em>I chose a 3D polytope</em>
</div>
<p>You can find thousands of additional plotting examples in the <a href="http://doc.sagemath.org/html/en/reference/index.html#graphics">Sage reference manual</a>.</p>
<h3 id="use-latex">Use LaTeX</h3>
<p>Next, let’s create a LaTeX document.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.53.29 AM.png" style="width:90%" />
<br />
<em>In +New, click LaTeX</em>
</div>
<p>A new LaTeX editor will appear. Edit source on the left, see the preview (on save) on the right. Double click
the preview to go to the corresponding point in the source, etc.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.53.55 AM.png" style="width:90%" />
<br />
<em>Edit a LaTeX document</em>
</div>
<h3 id="files">Files</h3>
<p>Click Files at the top and you’ll see that we’ve made a lot of files:</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.54.13 AM.png" style="width:90%" />
<br />
<em>Files in our Project</em>
</div>
<h3 id="full-emacs">Full Emacs</h3>
<p>For fun, let’s
click on the tab for your X11 desktop (the file ending in .x11), then type</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>emacs *.tex &
</code></pre></div></div>
<p>to open that same LaTeX file in Emacs. Then click the Emacs tab on the right.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 8.56.50 AM.png" style="width:90%" />
<br />
<em>Open our LaTeX file in classical Emacs in the X11 Desktop</em>
</div>
<p>You can hit control+c twice in Emacs to build the LaTeX file from there, and have full <a href="https://www.gnu.org/software/auctex/">AUCTeX support</a>.</p>
<h3 id="microsoft-visual-studio-code">Microsoft Visual Studio Code</h3>
<p><a href="https://code.visualstudio.com/">Visual Studio Code</a> is also included with cocalc-docker. I tried using the
<a href="https://leanprover.github.io/">LEAN proof assistant</a> in it and it works great.</p>
<p>In your X11 Desktop (file ending .x11 terminal), type <code class="highlighter-rouge">code a.lean &</code> and VS Code pops up.</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 10.39.32 AM.png" style="width:90%" />
<br />
<em>Using Microsoft Visual Studio Code's LEAN proof assistant mode</em>
</div>
<p>When you edit the file, it’ll offer to search the marketplace for a plugin to support LEAN.
It’ll install it with a click, and then you have LEAN fully working. Note that the LEAN
server itself is already preinstalled in CoCalc-docker,
and you can also just open a.lean directly from the Files
listing in CoCalc to use CoCalc’s own LEAN editor (or you
could install one in Emacs).</p>
<h3 id="draw-some-doodles">Draw some Doodles</h3>
<p>CoCalc-docker also includes <a href="https://inkscape.org/">Inkscape</a> and <a href="https://www.libreoffice.org/">Libreoffice</a>:</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 9.38.38 AM.png" style="width:90%" />
<br />
<em>Doodle in Inkscape and Libreoffice draw</em>
</div>
<h3 id="learn-how-to-write-python3-games">Learn how to write Python3 Games</h3>
<p>You can even <a href="https://www.pygame.org/wiki/GettingStarted">use pygame</a> to write and play Python3 Games:</p>
<div style="text-align: center;">
<img src="/img/cocalc-docker-gcp/Screenshot 2018-11-07 at 9.48.26 AM.png" style="width:90%" />
<br />
<em>Play a Game</em>
</div>
<p>NOTE: sound is currently not supported.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Thanks for giving Cocalc-Docker a spin. I hope you find
the wide range of collaborative web-based functionality
useful to supporting your teaching and research. Please
let me (William Stein) know what you think in the comments below, or
by emailing <a href="mailto:help@sagemath.com">help@sagemath.com</a>.</p>William SteinIntroduction CoCalc is a web application that lets you collaboratively use a large amount of free open source math and data related open source software. You can create collaborative Jupyter notebooks, edit LaTeX documents, use Terminals, use graphical Linux applications, create chatrooms, and much more. There’s extensive support for SageMath, Octave (a MATLAB clone), and R. See the docs. Cocalc-docker is a completely free and open source self contained version of CoCalc, which you can run on your own computer or cloud server. This post is about how to freely play around with running CoCalc-docker on Google Cloud Platform. If you scroll down and see all the cool things CoCalc can do, but don’t want to bother running your own server, make an account at CoCalc and use our hosted service, which has filesystem snapshots, a vast amount of preinstalled software (much more than cocalc-docker), and support. Sign up for Google Cloud Platform Click above to learn about Google Cloud Platform's free trial Create a Container Instance Where to run it? Choose a location near you: Choose where to run the container close to you, for optimal speed! Select to run a container directly (and configure your machine type) Click the checkbox next to “Deploy a container image to this VM instance.”, then put sagemathinc/cocalc in the blank below “Container image”. Also check the boxes next to buffer stdin and allocate a tty. You can also change the machine type, though the default will work. Container Image and Machine type Increase Base Image Size to at least 20GB! http://www.sagemath.org/ CRITICAL: increase the base image size to at least 20GB! The default of 10GB will fail. Allow the container to provide an HTTP/HTTPS web server Enable https and http access Make the instance pre-emptible (optional) If you are just playing around to test this out, open “Management, security, disks, networking, sole tenancy” and scroll down and set “Preemptibility” to On. This will make things way cheaper (using less of your free trial credits). This is especially useful if you wan to do a relatively quick but very CPU intensive parallel computation. Here’s how our cost estimate comes out so far, with preemptible on. It would be about four times as expensive without preemptible on. Pretty cheap... Of course the drawback of preemptible is that the machine will be killed within 24 hours. That’s fine for testing things out though. Click Create at the bottom to start creating your VM You'll see this line in your list of VM's when the instance is being created Watch the Serial Port Once the VM is created, click to open it, then click “Serial port 1 (console)” (or “Connect to serial console”), to watch the log as the machine boots up. Watch the Serial Port It takes at least 10 minutes to pull and decompress the sagemathinc/cocalc Docker image. If this fails, you probably forgot to increase the size of the boot disk from 10GB to 20GB (or more), in which case you should delete everything and start over. Wait at least 10 minutes until you see the above Determine the IP address Once your machine is running and the sagemathinc/cocalc image has been pulled and decompressed, find the external ip address of your machine, and open it in a new browser tab. In my case, I open https://35.227.184.91/. Do NOT choose the address that starts 10., since that is internal. Copy your IP address If this fails, you probably forgot to check the box next to “Allow HTTPS traffic”. Security warning Since the SSL cert in the Docker image is self-signed, you’ll get a warning. Click through it by clicking “ADVANCED”. Click ADVANCED Click Proceed... Create a new account on your personal CoCalc server Click to create an account... Create an account WARNING: Anybody who knows the IP address can make an account in the same way. There’s no secret token, and currenly no way to configure one with GCP Container Image. See this issue.CoCalc brings collaborative persistent graphical Linux applications to your browser, with integrated clipboard and HiDPI support2018-11-05T00:00:00+00:002018-11-05T00:00:00+00:00http://blog.cocalc.com/cocalc/2018/11/05/x11<h2 id="graphical-linux-applications-finally-come-to-cocalc">Graphical Linux Applications Finally Come to CoCalc!</h2>
<p>The goal of <a href="https://CoCalc.com">CoCalc</a> is to make all open source mathematics
and data science software easily available in your web
browser, in order to reduce the barriers to using open source
software for
teaching courses and doing research. For many years,
people have been requesting the ability to run standard
graphical Linux applications in CoCalc. Until now,
we have primarily focused on <a href="http://jupyter.org/">Jupyter notebooks</a>, and other
“native” web applications. However, we finally figured out
how to bring standard graphical Linux applications
to CoCalc, which will further help in our mission of removing
barriers to using free open source software in teaching and research.</p>
<div style="text-align: center;">
<img src="/img/x11/nteract-and-emacs-and-python-tk.png" style="width:100%" />
<br />
<em>Using <a href="https://nteract.io/">Nteract</a> (a desktop Jupyter client), Emacs, and Python Tk in CoCalc</em>
</div>
<p>CoCalc now has X11 graphical support, which lets you run any graphical Linux application in your browser. <em>You want to try this out</em>: Go to <a href="https://cocalc.com">CoCalc.com</a>, make an account, create a project, then click
<code class="highlighter-rouge">+New</code> and select <code class="highlighter-rouge">X11 Desktop</code>.</p>
<div style="text-align: center;">
<img src="/img/x11/x11-button.png" style="width:80%" />
<br />
<em>starting an X11 display in CoCalc</em>
</div>
<p>You’ll get an <a href="https://xtermjs.org/">xterm.js</a> terminal on the left and a blank desktop on the right.
Type <code class="highlighter-rouge">xclock&</code>
in the terminal then press enter, and you’ll see a tab appear on the right. Click it and
see a clock.</p>
<div style="text-align: center;">
<img src="/img/x11/xclock.png" style="width:90%" />
<br />
<em>Running XClock in CoCalc</em>
</div>
<p>Type <code class="highlighter-rouge">python3</code> in the terminal, and try something involving
<a href="https://docs.python.org/3.3/library/turtle.html?highlight=turtle">Turtle graphics</a>, and it will just work:</p>
<div style="text-align: center;">
<img src="/img/x11/turtle.png" style="width:90%" />
<br />
<em>Running XClock in CoCalc</em>
</div>
<p>Many other graphical programs work the same way,
including <a href="https://inkscape.org/">inkscape</a>, <a href="https://www.libreoffice.org/">libreoffice</a>,
<a href="https://octave.org/doc/interpreter/index.html">octave</a>, <a href="https://github.com/vim/vim">gvim</a>, <a href="https://www.gnu.org/software/emacs/">emacs</a>, <a href="https://wiki.gnome.org/Projects/GnomeShell">gnome-shell</a>,
<a href="https://code.visualstudio.com/">VS code</a>, <a href="https://nteract.io/">nteract</a>, <a href="https://swirlstats.com/">swirl</a>,
<a href="https://www.pygame.org/wiki/GettingStarted">PyGame</a>,
and <a href="https://github.com/sagemathinc/cocalc/wiki/x11-help#applications">much more</a>. Let us know what you want.</p>
<p>If you run anything that tries to display X11 graphics, it’ll also just pop up in the window on the right.</p>
<h2 id="cocalc-adds-persistence-collaboration-hidpiretina-support-and-full-clipboard-integration">CoCalc adds Persistence, Collaboration, HiDPI/Retina support, and Full Clipboard Integration</h2>
<p>Our support for X11 is based on the <a href="https://xpra.org/">Xpra remote display server</a>, with our own <a href="https://github.com/sagemathinc/cocalc/tree/master/src/smc-webapp/frame-editors/x11-editor/xpra">new Xpra client</a>, which is a complete rewrite of <a href="https://xpra.org/trac/wiki/Clients/HTML5">the official Xpra HTML5 client</a>. We have also integrated two core features of CoCalc:</p>
<ul>
<li>
<p><strong>Persistence.</strong> You can close your browser, then come back later to the exact same session (with the same applications running), as long as your CoCalc project is still running.</p>
</li>
<li>
<p><strong>Collaboration.</strong> Another person can open the same X11 desktop and interactively use the same programs you’re using. This is very helpful for video or text chats (supported via chat on the right), and other collaborative work.</p>
</li>
</ul>
<p>These features give CoCalc’s X11 support an edge over traditional X11 forwarding, for example via SSH. Xpra is also generally faster than X11 forwarding over SSH.</p>
<p>Our Xpra integration also looks crisp because it properly
handles HiDPI and Retina displays by appropriately rescaling
windows, and we provide zoom buttons to very easily
tweak the size to your liking.</p>
<p>Finally, there is full clipboard integration between your computer
and the remote X11 session (copy only works on Chrome 66+).</p>
<h2 id="screenshots">Screenshots</h2>
<h3 id="octave">Octave</h3>
<div style="text-align: center;">
<img src="/img/x11/octave-cli-2.jpg" style="width:90%" />
<br />
<em><a href="https://octave.org/doc/interpreter/index.html">GNU Octave</a> launched from a terminal with plot in X11</em>
</div>
<h3 id="libreoffice">LibreOffice</h3>
<div style="text-align: center;">
<img src="/img/x11/r-libreoffice.png" style="width:90%" />
<br />
<em>R command line (left) exports dataframe to .xlsx file, which we then display in <a href="https://www.libreoffice.org/">libreoffice</a>(right)</em>
</div>
<h3 id="octave-gui">Octave GUI</h3>
<div style="text-align: center;">
<img src="/img/x11/octave-gui-2.jpg" style="width:90%" />
<br />
<em>CoCalc running GUI version of <a href="https://octave.org/doc/interpreter/index.html">GNU Octave</a> </em>
</div>
<h3 id="saoimageds9">SAOImageDS9</h3>
<div style="text-align: center;">
<img src="/img/x11/ds9-cas-a.png" style="width:90%" />
<br />
<em>Astronomical imaging in CoCalc with <a href="http://ds9.si.edu/site/Home.html">SAOImageDS9</a></em>
</div>
<h3 id="inkscape">Inkscape</h3>
<div style="text-align: center;">
<img src="/img/x11/inkscape.png" style="width:90%" />
<br />
<em>Editing the CoCalc logo using [Inkscape](https://inkscape.org/), with chat on the side</em>
</div>
<h3 id="chrome">Chrome</h3>
<div style="text-align: center;">
<img src="/img/x11/chrome-in-chrome-in-chrome.png" style="width:90%" />
<br />
<em>Chrome in Chrome in Chrome ... (NOTE: Chrome and Firefox will sometimes crash due to security restrictions of Docker, unfortunately.)</em>
</div>
<h3 id="snappy">Snappy</h3>
<p>If you want to try something that uses <strong>interactive OpenGL 3D graphics</strong>, run <a href="https://www.math.uic.edu/t3m/SnapPy/index.html">snappy</a> (<code class="highlighter-rouge">sage -python -m snappy.app</code> and wait about 20 seconds), then type this into snappy</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M = Manifold("8_20"); M.browse()
</code></pre></div></div>
<p>and click on the Dirichlet tab of the window that pops up. It’s pretty fast, actually. Much faster than I expected.</p>
<div style="text-align: center;">
<img src="/img/x11/snappy.png" style="width:90%" />
<br />
<em>Studying the topology and geometry of 3-manifolds...</em>
</div>William Stein and Hal SnyderGraphical Linux Applications Finally Come to CoCalc! The goal of CoCalc is to make all open source mathematics and data science software easily available in your web browser, in order to reduce the barriers to using open source software for teaching courses and doing research. For many years, people have been requesting the ability to run standard graphical Linux applications in CoCalc. Until now, we have primarily focused on Jupyter notebooks, and other “native” web applications. However, we finally figured out how to bring standard graphical Linux applications to CoCalc, which will further help in our mission of removing barriers to using free open source software in teaching and research. Using Nteract (a desktop Jupyter client), Emacs, and Python Tk in CoCalc CoCalc now has X11 graphical support, which lets you run any graphical Linux application in your browser. You want to try this out: Go to CoCalc.com, make an account, create a project, then click +New and select X11 Desktop. starting an X11 display in CoCalc You’ll get an xterm.js terminal on the left and a blank desktop on the right. Type xclock& in the terminal then press enter, and you’ll see a tab appear on the right. Click it and see a clock. Running XClock in CoCalc Type python3 in the terminal, and try something involving Turtle graphics, and it will just work: Running XClock in CoCalcCollaborative Editing2018-10-11T00:00:00+00:002018-10-11T00:00:00+00:00http://blog.cocalc.com/2018/10/11/collaborative-editing<h1 id="collaborative-editing-in-cocalc-ot-crdt-or-something-else">Collaborative Editing in CoCalc: OT, CRDT, or something else?</h1>
<p><a href="https://news.ycombinator.com/item?id=18191867">This paper</a> about collaborative editing is on Hacker News today.
I also recently talked with <a href="https://github.com/sccolbert">Chris Colbert</a> about his new plans
to use a CRDT approach to collaborative editing
of Jupyter notebooks.
This has caused me to be very curious again about how <a href="https://cocalc.com">CoCalc’s</a> collaborative editing is related to the many algorithms and research around
the problem in the literature.</p>
<h2 id="the-collaborative-editing-problem">The Collaborative Editing Problem</h2>
<p>Protocols for collaborative editing are a venerable problem in computer
science, and there are probably over a hundred published
research papers on it. The basic setup, going back three decades, is that sync algorithms are supposed to have three properties, which I’ve stated in simplified
plain language below:</p>
<ul>
<li><strong>1. convergence:</strong> everybody’s document looks the same if they all take their hands off the keyboard for a while,</li>
<li><strong>2. causality preservation:</strong> if somebody else’s change is applied to your local version of the document, then you make a change, then your change is always applied after their change (in 1).</li>
<li><strong>3. intention preservation:</strong> the effect of applying a change operation is the same for everybody, as when it was initially done. (Yes, this is vague; it’s much harder to make
precise than 1 and 2.)</li>
</ul>
<p><strong>CoCalc has 1</strong>, of course; without that you’ve got nothing.</p>
<p><strong>CoCalc has 2</strong>, when people’s clocks are synced, because all patches you’ve applied have timestamp less than now (=time when making the patch).</p>
<p><strong>CoCalc does NOT have 3</strong>, for some meaning of 3. Patches are applied on a “best effort basis”. So instead of our changes being “insert the word ‘foo’ at position 7”, they are more vague, e.g., apply this patch with this context using these parameters to determine Levenshtein distance between strings. With intention preservation, if the operation is “insert word ‘foo’ at position 7”, definitely that’s exactly what happens whenever anybody does it (‘foo’ will appear in the document) – it does not depend at all on context. With <a href="https://github.com/google/diff-match-patch">diffmatchpatch</a> patches (which we use in CoCalc), the effect of the patch depends very much on the document you’re applying the patch to. If there is insufficient context, then ‘foo’ might not get inserted at all.</p>
<p>Similar remark apply to how I designed the structured object sync in CoCalc, which is used, e.g., for <a href="https://cocalc.com/doc/jupyter-notebook.html">CoCalc Jupyter Notebooks</a>;
it also applies patches on a best effort basis.</p>
<h2 id="ot--operational-transforms">OT = operational transforms</h2>
<p>This is a protocol that <em>in theory</em> has all of 1-3. Of course there
are many, many specific versions of OT. The hard part is ensuring 3,
and it can be complicated. The problem to be solved makes sense,
and it can be done. The details (and implementing
them) are certainly nontrivial to think about conceptually…
There’s many academic research papers on OT,
and it’s implemented (well) in many production
systems.</p>
<p>In OT, the data structure that defines the document is simple (e.g., just a text string), and the operations are simple, but applying them in a meaningful way is <em>very</em> hard. <a href="https://news.ycombinator.com/item?id=18191867">This paper on HN that I mentioned above</a> argues that OT
is much more popular in production systems than CRDT.</p>
<h2 id="crdt--commutative-replicated-data-type">CRDT = commutative replicated data type</h2>
<p>This also does 1-3. It sets everything up so the
data structure that defines the document is very complicated
(and <em>verbose</em>), but it’s always possible to merge documents in
a consistent way. What is difficult gets pushed to different
places in the protocol than OT, but it’s still quite hard,
and there are subtle issues involved with any non-toy implementation.</p>
<h2 id="what-about-cocalcs-approach">What about CoCalc’s approach…?</h2>
<p>CoCalc’s text editing does synchronization as follows. Each
user periodically computes a timestamped patch, then broadcasts
it to everybody else editing the same file. When patches arrive,
each user computes the current state of the document as the
result of applying all patches in timestamp order.
If everybody stops editing, then they all agree on the same
document.</p>
<p>This protocol satisfies 1 and 2, but not 3. The reason is that
patches are applied on a best-effort basis using the diff-match-patch
algorithm. For example, a patch made from deleting a single letter
in a document can, when applied to a different document end
up deleting multiple letters (or none). Basically, CoCalc replaces
all the very hard work needed for 3 that OT and CRDT’s have with a notion of applying patches on a “best
effort” basis. The behavior is well defined (because of the timestamps),
but may be surprising when multiple people do simultaneous nearby
edits in a document.</p>
<p>The paper says:</p>
<blockquote>
<p>“There are two basic ways to propagate local
edits: one is to propagate the edits as operations [12,38,50,51,73]; the other is to propagate the edits
as states [13]. Most real-time co-editors, including those based on OT and CRDT, have adopted
the operation approach for propagation for communication efficiency, among others. The operation
approach is assumed for all editors discussed in the rest of this paper”.</p>
</blockquote>
<p>Here [13] is <a href="https://neil.fraser.name/writing/sync/">N. Fraser’s paper on Differential Sync</a>. This was the sync
algorithm in the first version of CoCalc, and was the inspiration for
what CoCalc currently does.</p>
<p>In CoCalc, the data structure that defines the document is simple (just a text string, say), and the operations are less simple (computing diffs, defining patches), and applying them in a meaningful way is
somewhat difficult (it’s what the diffmatchpatch library does).
This approach is very easy to think about and generalize, since it
is self contained and a local problem. After all, I mostly described the algorithm in a single paragraph above!</p>
<p>In CoCalc, we compute diffs of arbitrary documents periodically,
much like how React.js DOM updates work.
This seems to not be needed in OT or CRDT, which instead track the
actual operations performed by users (i.e., type a character,
delete something).
Computing diffs has bad complexity in general, but very good
complexity in many cases that matter in practice (that’s
the trick behind React). Diffs involve <strong>observing state</strong> periodically, rather than tracking changes.</p>
<p>OT and CRDT really are solving a much harder problem than we solve. This is similar to how git uses the trick of “assume sha1 hashes don’t collide” to solve a much easier problem than the much harder problems other revision control systems like <a href="http://darcs.net:5001/">Darcs</a> solve.</p>
<h2 id="an-example-in-which-cocalc-violates-the-intention-preservation-requirement">An Example in which CoCalc violates the intention preservation requirement</h2>
<p>There is a nice example to illustrate how CoCalc fails for this third “user intention” requirement. This is called <em>“the TP2 puzzle”</em>. You can try the following in both <a href="https://cocalc.com">CoCalc</a> and <a href="https://www.overleaf.com/">Overleaf</a> (which probably does some OT algorithm):</p>
<ol>
<li>Type in some blank lines, then “abcd”, then blank lines</li>
<li>Open three windows on the doc you’re editing.</li>
<li>Disconnect your Internet</li>
<li>In each of the three window, make these changes, in order:
<ul>
<li>abcxd (put x after c)</li>
<li>abycd (put y before c)</li>
<li>acd (delete b)</li>
</ul>
</li>
<li>Reconnect and watch. The experts agree that the “correct” intention preserving convergent state is “aycxd” (which overleaf produces), but CoCalc will produce “acxd”.</li>
</ol>
<p>I do NOT consider this a <em>bug</em> in CoCalc – it’s doing exactly what is implemented, and what I as the author of the realtime sync system
intended.
The issue is that the patch to delete “b” has “a” and “cd” as surrounding
context, and if you look at how diffmatchpatch patch application works, this is a case where it just deletes everything inside the context.</p>
<p>Evidently, <a href="http://www.thinkbottomup.com.au/site/blog/Google_Wave_Intention_Preservation_Branching_Merging_and_TP2">Google Wave also had issues with TP2</a> because fully implementing OT is…</p>
<blockquote>
<p>“… hard! In fact, almost all published algorithms that claim to satisfy TP2 have been shown to be flawed.”</p>
</blockquote>
<h3 id="more-details">More details…</h3>
<p>The “famous” TP2 puzzle for CoCalc ends up like this (in at least 1 of the 6 possibilities!).</p>
<p>Start with</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>abc
</code></pre></div></div>
<p>then add an x and a y on either side of b, and delete b.</p>
<p>In one order, end up with</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>acxd
</code></pre></div></div>
<p>The patches are:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [[[[0,"--\n\n\nabc"],[1,"x"],[0,"d\n\n\n\n\n"]],5291,5291,14,15]]
[[[[0,"---\n\n\nab"],[1,"y"],[0,"cd\n\n\n\n\n"]],5290,5290,15,16]]
[[[[0,"\n---\n\n\na"],[-1,"b"],[0,"cd\n\n\n\n\n"]],5289,5289,16,15]]
</code></pre></div></div>
<p>Applying the “delete b” patch, also deletes the y:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apply_patch([[[[0,"abc"],[1,"x"],[0,"d"]],5291,5291,14,15]], 'abcd')
(2) ["abcxd", true]
apply_patch([[[[0,"ab"],[1,"y"],[0,"cd"]],5290,5290,15,16]], "abcxd")
(2) ["abycxd", true]
apply_patch([[[[0,"a"],[-1,"b"],[0,"cd"]],5289,5289,16,15]], "abycxd")
(2) ["acxd", true]
</code></pre></div></div>
<p>Looking at the <a href="https://github.com/google/diff-match-patch/blob/master/javascript/diff_match_patch_uncompressed.js#L1880">source code of diffmatchpatch</a>, this is just what DMP does. If there is a lot more badness and the strings are bigger, it’ll refuse to delete. It really is a sort of “best effort application of patches” with parameters and heuristics; no magic there.</p>
<h3 id="conclusion">Conclusion</h3>
<p>I viewed the problem of realtime synchronized
editing of text to be primarily one of <strong>synchronization</strong>. The
goal is that if multiple people are editing a file
at once, then they take their hands of the keyboard,
everybody quickly converges to looking at the same document.
Moreover, that document should reflect what people
are typing – it should not be random gibberish!
By relaxing condition 3 “intention preservation”, which
is hard to rigorously define anyways and very hard to satisfy,
we obtain a very simple algorithm to implement and reason
about. So that’s what CoCalc uses. I had battled
for a while with other approaches, and decided
on switching to this very simple approach, since I wanted
something that worked solidly in practice and would be
easy to implement and extend.</p>SageMath, Inc.Collaborative Editing in CoCalc: OT, CRDT, or something else? This paper about collaborative editing is on Hacker News today. I also recently talked with Chris Colbert about his new plans to use a CRDT approach to collaborative editing of Jupyter notebooks. This has caused me to be very curious again about how CoCalc’s collaborative editing is related to the many algorithms and research around the problem in the literature. The Collaborative Editing Problem Protocols for collaborative editing are a venerable problem in computer science, and there are probably over a hundred published research papers on it. The basic setup, going back three decades, is that sync algorithms are supposed to have three properties, which I’ve stated in simplified plain language below: 1. convergence: everybody’s document looks the same if they all take their hands off the keyboard for a while, 2. causality preservation: if somebody else’s change is applied to your local version of the document, then you make a change, then your change is always applied after their change (in 1). 3. intention preservation: the effect of applying a change operation is the same for everybody, as when it was initially done. (Yes, this is vague; it’s much harder to make precise than 1 and 2.) CoCalc has 1, of course; without that you’ve got nothing. CoCalc has 2, when people’s clocks are synced, because all patches you’ve applied have timestamp less than now (=time when making the patch). CoCalc does NOT have 3, for some meaning of 3. Patches are applied on a “best effort basis”. So instead of our changes being “insert the word ‘foo’ at position 7”, they are more vague, e.g., apply this patch with this context using these parameters to determine Levenshtein distance between strings. With intention preservation, if the operation is “insert word ‘foo’ at position 7”, definitely that’s exactly what happens whenever anybody does it (‘foo’ will appear in the document) – it does not depend at all on context. With diffmatchpatch patches (which we use in CoCalc), the effect of the patch depends very much on the document you’re applying the patch to. If there is insufficient context, then ‘foo’ might not get inserted at all. Similar remark apply to how I designed the structured object sync in CoCalc, which is used, e.g., for CoCalc Jupyter Notebooks; it also applies patches on a best effort basis. OT = operational transforms This is a protocol that in theory has all of 1-3. Of course there are many, many specific versions of OT. The hard part is ensuring 3, and it can be complicated. The problem to be solved makes sense, and it can be done. The details (and implementing them) are certainly nontrivial to think about conceptually… There’s many academic research papers on OT, and it’s implemented (well) in many production systems. In OT, the data structure that defines the document is simple (e.g., just a text string), and the operations are simple, but applying them in a meaningful way is very hard. This paper on HN that I mentioned above argues that OT is much more popular in production systems than CRDT. CRDT = commutative replicated data type This also does 1-3. It sets everything up so the data structure that defines the document is very complicated (and verbose), but it’s always possible to merge documents in a consistent way. What is difficult gets pushed to different places in the protocol than OT, but it’s still quite hard, and there are subtle issues involved with any non-toy implementation. What about CoCalc’s approach…? CoCalc’s text editing does synchronization as follows. Each user periodically computes a timestamped patch, then broadcasts it to everybody else editing the same file. When patches arrive, each user computes the current state of the document as the result of applying all patches in timestamp order. If everybody stops editing, then they all agree on the same document. This protocol satisfies 1 and 2, but not 3. The reason is that patches are applied on a best-effort basis using the diff-match-patch algorithm. For example, a patch made from deleting a single letter in a document can, when applied to a different document end up deleting multiple letters (or none). Basically, CoCalc replaces all the very hard work needed for 3 that OT and CRDT’s have with a notion of applying patches on a “best effort” basis. The behavior is well defined (because of the timestamps), but may be surprising when multiple people do simultaneous nearby edits in a document. The paper says: “There are two basic ways to propagate local edits: one is to propagate the edits as operations [12,38,50,51,73]; the other is to propagate the edits as states [13]. Most real-time co-editors, including those based on OT and CRDT, have adopted the operation approach for propagation for communication efficiency, among others. The operation approach is assumed for all editors discussed in the rest of this paper”. Here [13] is N. Fraser’s paper on Differential Sync. This was the sync algorithm in the first version of CoCalc, and was the inspiration for what CoCalc currently does. In CoCalc, the data structure that defines the document is simple (just a text string, say), and the operations are less simple (computing diffs, defining patches), and applying them in a meaningful way is somewhat difficult (it’s what the diffmatchpatch library does). This approach is very easy to think about and generalize, since it is self contained and a local problem. After all, I mostly described the algorithm in a single paragraph above! In CoCalc, we compute diffs of arbitrary documents periodically, much like how React.js DOM updates work. This seems to not be needed in OT or CRDT, which instead track the actual operations performed by users (i.e., type a character, delete something). Computing diffs has bad complexity in general, but very good complexity in many cases that matter in practice (that’s the trick behind React). Diffs involve observing state periodically, rather than tracking changes. OT and CRDT really are solving a much harder problem than we solve. This is similar to how git uses the trick of “assume sha1 hashes don’t collide” to solve a much easier problem than the much harder problems other revision control systems like Darcs solve. An Example in which CoCalc violates the intention preservation requirement There is a nice example to illustrate how CoCalc fails for this third “user intention” requirement. This is called “the TP2 puzzle”. You can try the following in both CoCalc and Overleaf (which probably does some OT algorithm): Type in some blank lines, then “abcd”, then blank lines Open three windows on the doc you’re editing. Disconnect your Internet In each of the three window, make these changes, in order: abcxd (put x after c) abycd (put y before c) acd (delete b) Reconnect and watch. The experts agree that the “correct” intention preserving convergent state is “aycxd” (which overleaf produces), but CoCalc will produce “acxd”. I do NOT consider this a bug in CoCalc – it’s doing exactly what is implemented, and what I as the author of the realtime sync system intended. The issue is that the patch to delete “b” has “a” and “cd” as surrounding context, and if you look at how diffmatchpatch patch application works, this is a case where it just deletes everything inside the context. Evidently, Google Wave also had issues with TP2 because fully implementing OT is… “… hard! In fact, almost all published algorithms that claim to satisfy TP2 have been shown to be flawed.” More details… The “famous” TP2 puzzle for CoCalc ends up like this (in at least 1 of the 6 possibilities!). Start with abc then add an x and a y on either side of b, and delete b. In one order, end up with acxd The patches are: [[[[0,"--\n\n\nabc"],[1,"x"],[0,"d\n\n\n\n\n"]],5291,5291,14,15]] [[[[0,"---\n\n\nab"],[1,"y"],[0,"cd\n\n\n\n\n"]],5290,5290,15,16]] [[[[0,"\n---\n\n\na"],[-1,"b"],[0,"cd\n\n\n\n\n"]],5289,5289,16,15]] Applying the “delete b” patch, also deletes the y: apply_patch([[[[0,"abc"],[1,"x"],[0,"d"]],5291,5291,14,15]], 'abcd') (2) ["abcxd", true] apply_patch([[[[0,"ab"],[1,"y"],[0,"cd"]],5290,5290,15,16]], "abcxd") (2) ["abycxd", true] apply_patch([[[[0,"a"],[-1,"b"],[0,"cd"]],5289,5289,16,15]], "abycxd") (2) ["acxd", true] Looking at the source code of diffmatchpatch, this is just what DMP does. If there is a lot more badness and the strings are bigger, it’ll refuse to delete. It really is a sort of “best effort application of patches” with parameters and heuristics; no magic there.Where does CoCalc come from?2018-09-10T00:00:00+00:002018-09-10T00:00:00+00:00http://blog.cocalc.com/cocalc/2018/09/10/where-is-cocalc-from<h4 id="meet-the-team-and-company-that-provides-cocalc">Meet the team and company that provides CoCalc.</h4>
<h2 id="cocalc-origins">CoCalc Origins</h2>
<p>Prior to CoCalc, William Stein spent 15 years teaching and doing research using mathematical software at Berkeley, Harvard, UCSD, and Univ of Washington. Based on this experience, he launched the <a href="https://cocalc.com">CoCalc web application</a> in April 2013, under the name SageMathCloud, with the mission <strong>to make it very
easy to collaboratively use free open source
mathematics and data science software in classes and research</strong>.</p>
<p>After over 5 years of extremely active development, CoCalc
is now a modern web application that provides
collaborative access to most free open source
technical software, including LaTeX, Jupyter Notebooks,
the Python numerical ecosystem, the R statistics software, and SageMath.
Thus CoCalc brings together the work of thousands
of contributors to open source software under one
roof, which is easily accessible from your web
browser.</p>
<h2 id="the-company">The Company</h2>
<p>Sagemath Inc., the company behind CoCalc, is an independent Seattle based tech startup founded by William Stein in Feb 2015. The
mission of SageMath, Inc. is to <strong>be a
sustainable company whose interests are
strongly aligned with
the open source mathematical software community</strong>.</p>
<p>SageMath, Inc. provides new sources of support
for open source mathematics software, by contributing to those projects
and making them more accessible.
Though SageMath, Inc. continues to grow substantially,
it does not seek venture capital investment and will
not be purchased by a large tech company. The
company has greatly benefited from support by
sophisticated investors with a love of mathematics.
(If you would like to be such an investor, please
<a href="contact@sagemath.com">contact us</a>.)</p>
<h2 id="who-we-are">Who We Are</h2>
<p>There are currently four people–William, Harald, John, and Hal–who work fulltime on CoCalc (though many others have contributed to CoCalc):</p>
<ul>
<li><a href="https://wstein.org/">William Stein</a> (Seattle)</li>
<li><a href="http://harald.schil.ly/">Harald Schilly</a> (Vienna, Austria)</li>
<li><a href="https://johnjeng.com/personal/">John Jeng</a> (Seattle)</li>
<li><a href="https://twitter.com/haldroid">Hal Snyder</a> (Chicago)</li>
</ul>
<h3 id="william-stein">William Stein</h3>
<p><img src="/img/who-is-cocalc/wsclip.png" alt="William Stein" style="width:20%" /></p>
<p>William is a pure mathematician specializing in number theory, who has worked on all aspects of frontend and backend software development related to both CoCalc and SageMath.</p>
<ul>
<li>2000: Mathematics Ph.D. from UC Berkeley in <em>number theory</em></li>
<li>2000-2005: Postdoc in mathematics at Harvard</li>
<li>2004: Founded <a href="http://www.sagemath.org/">SageMath</a></li>
<li>2006-current: Tenured Professor of Mathematics at Univ of Washington</li>
<li>2007: Started <a href="http://cython.org/">Cython</a></li>
<li>2016-current: <a href="https://wstein.org/talks/2016-06-sage-bp/bp.pdf">On leave from academia</a> to work fulltime on SageMath, Inc.</li>
<li>Published <a href="https://wstein.org/books/">several books</a> and <a href="https://wstein.org/papers/">many papers</a></li>
<li><a href="https://vertramp.org/">Skateboards his vert ramp</a> and goes hiking with <a href="https://share.cocalc.com/share/806e474f-4729-4f61-b85f-2ede9f961932/share/random/.html/IMG-1243-medium.jpg">his blue healer puppy</a>.</li>
<li><a href="https://github.com/williamstein" target="_blank"><img src="/img/icons/github.svg" style="width:28px;height:28px;border:0;vertical-align: middle;" /></a>
<a href="https://twitter.com/wstein389" target="_blank"><img src="/img/icons/twitter.svg" style="width:28px;height:28px;border:0;vertical-align: middle;" /></a></li>
</ul>
<h3 id="harald-schilly">Harald Schilly</h3>
<p><img src="/img/who-is-cocalc/hsyclip.png" alt="Harald Schilly" style="width:20%" /></p>
<p>Harald is an applied mathematician, with expertise in Linux, Python, Java, Javascript, Kubernetes, and system monitoring. He does the majority of CoCalc user support,
and manages the massive installation of open source
software that CoCalc projects use.</p>
<ul>
<li>Started working on CoCalc in 2015, after working on a Ph.D. at Univ of Vienna</li>
<li>Occasionally visits nature, likes cooking, and knows all about cryptocurrencies</li>
<li><a href="https://github.com/haraldschilly" target="_blank"><img src="/img/icons/github.svg" style="width:28px;height:28px;border:0;vertical-align: middle;" /></a>
<a href="https://twitter.com/Ha_Sch" target="_blank"><img src="/img/icons/twitter.svg" style="width:28px;height:28px;border:0;vertical-align: middle;" /></a>
<a href="https://www.linkedin.com/in/harald-schilly-519b2813/" target="_blank"><img src="/img/icons/linkedin.svg" style="width:28px;height:28px;border:0;vertical-align: middle;" /></a></li>
</ul>
<h3 id="john-jeng">John Jeng</h3>
<p><img src="/img/who-is-cocalc/j3clip.png" alt="John Jeng" style="width:20%" /></p>
<p>John does frontend software development (React, Javascript, UI, build tooling), and manages the company HR, accounting and general business processes.</p>
<ul>
<li>Started working on CoCalc in 2015</li>
<li>VP of Operations</li>
<li>2016, Mathematics BS from Univ of Washington</li>
<li>Personal trainer, unicyclist, board game specialist</li>
<li><a href="https://www.linkedin.com/in/jengjohn/" target="_blank"><img src="/img/icons/linkedin.svg" style="width:28px;height:28px;border:0;vertical-align: middle;" /></a></li>
</ul>
<h3 id="hal-snyder">Hal Snyder</h3>
<p><img src="/img/who-is-cocalc/halclip.png" alt="Hal Snyder" style="width:20%" /></p>
<p>Hal spent decades working at small and large tech companies, and now works on CoCalc’s testing, customer relationship management, user support, advertising, and Python development.</p>
<ul>
<li>Started working on CoCalc in 2016</li>
<li>1973, Mathematics MS from Univ of Chicago</li>
<li>1981, M.D, Northwestern University</li>
<li>Software developer (several decades)</li>
<li>Previous employment at Expedia and several other tech companies</li>
<li>Enjoys vegan cooking and taking massive open online courses (MOOCs), having completed more than 70 of them</li>
<li><a href="https://github.com/DrXyzzy/" target="_blank"><img src="/img/icons/github.svg" style="width:28px;height:28px;border:0;vertical-align: middle;" /></a>
<a href="https://twitter.com/HalDroid" target="_blank"><img src="/img/icons/twitter.svg" style="width:28px;height:28px;border:0;vertical-align: middle;" /></a>
<a href="https://www.linkedin.com/in/hsnyder/" target="_blank"><img src="/img/icons/linkedin.svg" style="width:28px;height:28px;border:0;vertical-align: middle;" /></a></li>
</ul>
<h3 id="other-contributors">Other Contributors</h3>
<p>Past significant contributors include: Greg Bard, Rob Beezer, Keith Clawson, Tim Clemans, Andy Huchala, Jon Lee, Simon Luu, Nicholas Ruhland, Travis Scholl, and Todd Zimmerman.</p>
<p>Much of CoCalc is open source.
Anyone can review the code (and offer pull requests!)
<a href="https://github.com/sagemathinc/cocalc">on GitHub</a>.</p>
<h2 id="where-cocalc-runs">Where CoCalc Runs</h2>
<p>The <a href="https://cocalc.com">public CoCalc.com web application</a>
often has over <a href="https://share.cocalc.com/share/7561f68d-3d97-4530-b97e-68af2fb4ed13/stats.html">5000 active users a day</a> served via a scalable and battle tested architecture built on <a href="https://kubernetes.io/">Kubernetes</a> and deployed
on Google Cloud Platform.
This scalable backend is proprietary;
contact us at <a href="mailto:help@sagemath.com">help@sagemath.com</a> if you are interested in running it on your infrastructure.</p>
<p>There is also a free and open source <a href="https://github.com/sagemathinc/cocalc/wiki/CoCalc-Docker-Image">CoCalc Docker image</a>, suitable for easily running your own private
CoCalc server on your computer.
This image, currently about 10 GB, does not contain all the nearly 300 GB of open-source software available
at <a href="https://cocalc.com">https://cocalc.com</a>.
However, it fully supports collaboratively running Jupyter, Python, SageMath, R, Julia, Octave, and editing LaTeX documents.</p>William Stein and Hal SnyderMeet the team and company that provides CoCalc. CoCalc Origins Prior to CoCalc, William Stein spent 15 years teaching and doing research using mathematical software at Berkeley, Harvard, UCSD, and Univ of Washington. Based on this experience, he launched the CoCalc web application in April 2013, under the name SageMathCloud, with the mission to make it very easy to collaboratively use free open source mathematics and data science software in classes and research. After over 5 years of extremely active development, CoCalc is now a modern web application that provides collaborative access to most free open source technical software, including LaTeX, Jupyter Notebooks, the Python numerical ecosystem, the R statistics software, and SageMath. Thus CoCalc brings together the work of thousands of contributors to open source software under one roof, which is easily accessible from your web browser.Dash with CoCalc2018-06-08T00:00:00+00:002018-06-08T00:00:00+00:00http://blog.cocalc.com/cocalc/python/2018/06/08/dash-with-cocalc<p><em>NOTE:</em> This blog posting has been updated. See <a href="../../../2021/05/03/dash-with-cocalc.html">Dash with CoCalc (2021 version)</a> for the updated article.</p>
<p>Create interactive data visualizations for collaborators in your CoCalc projects using <a href="https://plot.ly/products/dash/">Dash</a>.</p>
<p>Dash is an open-source framework to create web applications with Python.
With CoCalc’s <a href="https://github.com/sagemathinc/cocalc/wiki/HTTPWebserver">HTTPWebserver</a> capability, you can run a Dash application from inside a CoCalc project.</p>
<div style="text-align: center;">
<img src="/img/dash/dash-demo.png" style="width:80%" />
<br />
<em>Dash application running in a CoCalc project</em>
</div>
<p>We’ll step through a working program based on the first example in the <a href="https://dash.plot.ly/getting-started">Dash Tutorial</a>,
with a few additions for configuring the HTTP server.
Code for the <a href="https://cocalc.com/share/db982efa-e439-4e2d-933b-7c7011c6b21a/DASH/dash-demo.py?viewer=share">completed CoCalc example</a> is viewable on the share server.</p>
<p>Start by installing and updating Python3 packages in your project from a .term file:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip3 install --user dash
pip3 install --user dash_core_components
pip3 install --user dash_html_components
pip3 install --user --upgrade plotly
</code></pre></div></div>
<p>Copy the first example python file, “Hello Dash”, from the tutorial Python excerpt <code class="highlighter-rouge">app.py</code> into a .py file in your project. The code is short enough that you can use copy and
paste.</p>
<p>Insert the following three code snippets just above the test for <code class="highlighter-rouge">__name__ == '__main__'</code>.</p>
<p>Set the HTTP port number for your Dash web app. You can probably use the default port number of 8050, but you will
need to specify the port number in the prefix regardless.
Our example uses 9990.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>port = 9990
</code></pre></div></div>
<p>Get your project ID from the environment variable <code class="highlighter-rouge">COCALC_PROJECT_ID</code>. That is the specially-formatted
hex code you see in the browser URL when
logged into your project, and displayed the
Project Settings tab under Project Control.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import os
cocalc_project_id = os.environ['COCALC_PROJECT_ID']
</code></pre></div></div>
<p>Configure <code class="highlighter-rouge">requests_pathname_prefix</code> as indicated
in the <a href="https://github.com/sagemathinc/cocalc/wiki/HTTPWebserver">HTTPWebserver</a> wiki page:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pfx = "/{}/server/{}/".format(cocalc_project_id, port)
app.config.requests_pathname_prefix = pfx
</code></pre></div></div>
<p>In the final line, display the URL for your server
and override default port number and
host in the call to <code class="highlighter-rouge">app.run_server()</code>:</p>
<pre>
if __name__ == '__main__':
print("browse to https://cocalc.com{}".format(pfx))
app.run_server(debug=True, port=port, host='0.0.0.0')
</pre>
<p>Back in your .term session, run the program. For example,
if the file you just edited is <code class="highlighter-rouge">app.py</code>, do this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python3 app.py
</code></pre></div></div>
<p>Copy the URL that is displayed by the program.
Open a new browser tab and paste in the URL. You will see the “Hello Dash” demo.</p>
<p>Note: Because of the way CoCalc security works, only collaborators on the given project can access the webserver. Our proxy server will reject all other requests.</p>
<p>More examples are available in the <a href="https://dash.plot.ly/">Dash User Guide</a> and <a href="https://medium.com/@plotlygraphs/introducing-dash-5ecf7191b503">Introducing Dash</a>.</p>Hal SnyderNOTE: This blog posting has been updated. See Dash with CoCalc (2021 version) for the updated article. Create interactive data visualizations for collaborators in your CoCalc projects using Dash. Dash is an open-source framework to create web applications with Python. With CoCalc’s HTTPWebserver capability, you can run a Dash application from inside a CoCalc project. Dash application running in a CoCalc projectUse CoCalc to Learn How to Program2018-05-29T00:00:00+00:002018-05-29T00:00:00+00:00http://blog.cocalc.com/cocalc/python/r/2018/05/29/learn-to-program<p>If you are new to coding, CoCalc makes it easy to get started.
All you need is a web browser and an Internet connection.
You don’t have to install software on your computer to
start learning Python, R, Julia, and other leading
open-source languages.</p>
<div style="text-align: center;">
<img src="/img/learn-to-program/r-exo-3.png" style="width:80%" />
<br />
<em>learning R by example</em>
</div>
<p>Once you have decided to start learning, what do you do next? Here are some suggestions:</p>
<ul>
<li>
<p>It helps to have a simple programming task as a goal that you can approach in small steps.
But if you’re not sure where to start, you can open a project and click (+)New,
then scroll down to “Library” and view the list of topics.
If you click on a topic, for example “First Steps in CoCalc”,
a description and small image appears.
Click “Get a copy” to have a copy of the material loaded into your project,
where you can step through the instructions.
There’s more information in the <a href="http://blog.sagemath.com/cocalc/2018/03/06/cocalc-library.html">CoCalc blog</a>.</p>
</li>
<li>
<p>Find your support system. Whether it’s in person or online,
it helps to have study pals to discuss questions and share ideas.
There are links for online support for specific languages at the CoCalc
wiki <a href="https://github.com/sagemathinc/cocalc/wiki/Support">Support page</a>.</p>
</li>
<li>
<p>If you get stuck, try the <a href="http://blog.sagemath.com/cocalc/2018/04/05/assistant.html">Assistant</a>
button at upper right in Jupyter notebooks and Sage worksheets.
But also recognize that getting stuck can be a normal part of the learning process,
and the answer may come to you after a short while.</p>
</li>
<li>
<p>If something puzzles you about how CoCalc works, click the Help button.
If your question involves a specific file, click on Help after navigating to that file.
We love getting feedback!</p>
</li>
</ul>Hal SnyderIf you are new to coding, CoCalc makes it easy to get started. All you need is a web browser and an Internet connection. You don’t have to install software on your computer to start learning Python, R, Julia, and other leading open-source languages. learning R by exampleEmbedding CoCalc in Your Application2018-05-21T00:00:00+00:002018-05-21T00:00:00+00:00http://blog.cocalc.com/cocalc/2018/05/21/embed-cocalc<p>Add scientific computing to
any online training platform
by embedding CoCalc.</p>
<p>Embedding CoCalc into an online learning platform or <a href="https://en.wikipedia.org/wiki/Learning_management_system">learning management system</a> (LMS) adds:</p>
<ul>
<li>programming environments for today’s scientific programming languages</li>
<li>thousands of <a href="https://cocalc.com/doc/software.html">installed software packages and libraries</a>, updated regularly</li>
<li>interactive computing using Jupyter notebooks and Sage worksheets</li>
<li>real time collaboration</li>
<li><a href="http://blog.cocalc.com/latex/2017/02/06/smc-for-latex.html">mathematical typesetting</a> with LaTeX</li>
<li>scalable computing resources - allocate among students as needed</li>
<li><a href="https://tutorial.cocalc.com">distribute, collect, and grade</a> class assignments</li>
</ul>
<div style="text-align: center;">
<img src="/img/embed/cc-in-iframe-3.png" style="width:80%" />
</div>
<p>For readers unfamiliar with the platform, the next two sections introduce two key features of CoCalc, notebooks and markdown.</p>
<h2 id="jupyter-notebooks">Jupyter Notebooks</h2>
<p>These interactive documents mix executable code cells with cells for Markdown text (see below).</p>
<p>A notebook contains headers and sections, text with explanations, and code with plots.
Creating an assignment can be as simple as leaving selected code cells empty.</p>
<p>Multiple notebooks per topic are supported.
Splitting a topic keeps the
scope of a learning session manageable and reduces the
confusion that can arise from large numbers of variables.</p>
<p>CoCalc users can choose from a growing list of tutorial notebooks and texts in the <a href="http://blog.cocalc.com/cocalc/2018/03/06/cocalc-library.html">CoCalc library</a>, which allows files to be copied directly into a CoCalc project.</p>
<h2 id="markdown">Markdown</h2>
<p>Markdown is a simple way of formatting text. It is easy to learn
and converts readily into other online formats such as HTML.
There are several descriptions online, including the
<a href="http://commonmark.org/help/">one-page summary at CommonMark</a>,
which also includes a link to a brief tutorial.</p>
<p>Markdown is used in Jupyter notebooks and CoCalc worksheets for
explanatory text, which can include mathematical expressions
typeset in LaTeX.
A typical notebook is an executable narrative
consisting of alternating markdown and code cells.</p>
<h2 id="cocalc-pricing">CoCalc Pricing</h2>
<p>CoCalc pricing is based on upgrade packages that include network access, disk space, memory, etc. You can apply upgrades to any project on which you are an owner or collaborator. Upgrades are available
under two types of plan:</p>
<ul>
<li>Personal subscriptions. Can be purchased by the month or the year.
A subscriber can distribute upgrades to all projects he or she is a collaborator on.</li>
<li>Course packages. Available for 1 week, 4 months (a semester),
or a full year. These do not auto-renew.</li>
</ul>
<p>https://cocalc.com/policies/pricing.html</p>
<h2 id="the-api">The API</h2>
<p>The basic setup is to get an API key from your account page, and then issue calls to CoCalc. You can create accounts and projects, and coordinate everything using your own system and database.
The CoCalc website has an <a href="https://cocalc.com/doc/api.html">API reference</a> which documents each call and provides examples. The CoCalc team will work with customers to resolve any issues that arise.</p>
<p><a href="https://www.minerva.kgi.edu/">Minerva Schools at KGI</a> are <a href="http://blog.cocalc.com/jupyter/api/2018/03/01/minerva-and-cocalc.html">using the CoCalc API</a> to enable their students to work in Jupyter notebooks embedded inside an iframe on their website.</p>
<h2 id="iframe-urls">IFrame URLs</h2>
<p>You may want to remove the upper menu bars when embedding a CoCalc file in your application. In that case, there is a special syntax for the URL to the file.
The word “files” appears within the URL and “?session=&fullscreen” is appended to the URL.</p>
<p>Here’s an example:</p>
<p>https://cocalc.com/projects/cd3c25e4-5fbd-439b-9604-6011584af918/files/EMBED/batman-python.ipynb?session=&fullscreen</p>
<p>To prevent exiting fullscreen mode, append “=kiosk”.</p>
<p>https://cocalc.com/projects/cd3c25e4-5fbd-439b-9604-6011584af918/files/EMBED/batman-python.ipynb?session=&fullscreen=kiosk</p>
<h2 id="conclusion">Conclusion</h2>
<p>If you already have an online learning platform and want to include the scientific computing features that CoCalc has to offer, you can use the pointers above to get started.
To learn more, contact CoCalc support via
<a href="mailto:help@cocalc.com?subject=embed CoCalc">email</a>
or visit <a href="https://cocalc.com/help?session=">the CoCalc help page online</a>.</p>Harald Schilly and Hal SnyderAdd scientific computing to any online training platform by embedding CoCalc. Embedding CoCalc into an online learning platform or learning management system (LMS) adds: programming environments for today’s scientific programming languages thousands of installed software packages and libraries, updated regularly interactive computing using Jupyter notebooks and Sage worksheets real time collaboration mathematical typesetting with LaTeX scalable computing resources - allocate among students as needed distribute, collect, and grade class assignmentsExamples Assistant2018-04-05T00:00:00+00:002018-04-05T00:00:00+00:00http://blog.cocalc.com/cocalc/2018/04/05/assistant<p><a href="https://cocalc.com">CoCalc</a> wants you to fully accomplish your computational work online.
To archive this goal,
CoCals has to provide a reliable service, offer and maintain the software you need, and package this in a powerful interface.</p>
<p>The new <strong>“Assistant”</strong> is one of the latest additions to the interface.
Its goal is to help you by offering a <strong>curated set of annotated code snippets</strong>.</p>
<div style="text-align: center;">
<img src="/img/assistant/cocalc-assistant.png" style="width:80%" />
</div>
<p>Open it from within a Sage Worksheet or Jupyter Notebook.</p>
<div style="text-align: center">
<i>Sage Worksheets</i>
<br />
<img src="/img/assistant/assistant-sagews.png" style="width:50%;" />
</div>
<p><br /></p>
<div style="text-align: center">
<i>Jupyter Notebooks</i>
<br />
<img src="/img/assistant/assistant-jupyter.png" style="width:50%;" />
</div>
<p>You can browse through the categories, explore the collected functionalities,
or search for certain keywords.</p>
<p>One click, and it inserts the selected code snippet with an accompanying description right where you are in your document.
Then, you can <em>inspect</em>, <em>modify</em> and <em>learn</em> from the example right in front of you.</p>
<p>All the code examples are available at <a href="https://github.com/sagemathinc/cocalc-assistant">github.com/sagemathinc/cocalc-assistant</a> and we’re happy to receive contributions.</p>Harald SchillyCoCalc wants you to fully accomplish your computational work online. To archive this goal, CoCals has to provide a reliable service, offer and maintain the software you need, and package this in a powerful interface. The new “Assistant” is one of the latest additions to the interface. Its goal is to help you by offering a curated set of annotated code snippets.