Petko's Coding Bloghttps://pminkov.github.io/blog/2023-10-08T15:36:00-07:00Learning Parquet with DuckDB2023-10-08T15:36:00-07:002023-10-08T15:36:00-07:00Petko Minkovtag:pminkov.github.io,2023-10-08:/blog/learning-parquet-with-duckdb.html<p>I was interested in learning more about <a href="https://parquet.apache.org/">Parquet</a>
and picked DuckDB as a tool to explore Parquet datasets.</p>
<p>It wasn't easy to find good Parquet datasets, but I found two blog posts from MotherDuck
(post <a href="https://motherduck.com/blog/exploring-stackoverflow-with-duckdb-on-motherduck-1/">one</a>
and <a href="https://motherduck.com/blog/exploring-stackoverflow-with-duckdb-on-motherduck-2/">two</a>)
which helped with that. These posts contain links to a Stack Overflow …</p><p>I was interested in learning more about <a href="https://parquet.apache.org/">Parquet</a>
and picked DuckDB as a tool to explore Parquet datasets.</p>
<p>It wasn't easy to find good Parquet datasets, but I found two blog posts from MotherDuck
(post <a href="https://motherduck.com/blog/exploring-stackoverflow-with-duckdb-on-motherduck-1/">one</a>
and <a href="https://motherduck.com/blog/exploring-stackoverflow-with-duckdb-on-motherduck-2/">two</a>)
which helped with that. These posts contain links to a Stack Overflow dataset, which you
can easily explore with DuckDB.</p>
<p>To do what's described in this blog post, you'll need to install the AWS CLI and DuckDB. To use the AWS CLI
you'll need an AWS account. Once you're done with that, you can immediately start
to explore the datasets:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>aws<span class="w"> </span>s3<span class="w"> </span>ls<span class="w"> </span>s3://us-prd-motherduck-open-datasets/stackoverflow/parquet/2023-05/<span class="w"> </span>--human-readable
<span class="m">2023</span>-07-15<span class="w"> </span><span class="m">07</span>:01:48<span class="w"> </span><span class="m">0</span><span class="w"> </span>Bytes<span class="w"> </span>
<span class="m">2023</span>-07-15<span class="w"> </span><span class="m">11</span>:16:11<span class="w"> </span><span class="m">517</span>.2<span class="w"> </span>MiB<span class="w"> </span>badges.parquet
<span class="m">2023</span>-07-15<span class="w"> </span><span class="m">11</span>:16:11<span class="w"> </span><span class="m">6</span>.9<span class="w"> </span>GiB<span class="w"> </span>comments.parquet
<span class="m">2023</span>-07-15<span class="w"> </span><span class="m">11</span>:16:11<span class="w"> </span><span class="m">163</span>.3<span class="w"> </span>MiB<span class="w"> </span>post_links.parquet
<span class="m">2023</span>-08-16<span class="w"> </span><span class="m">07</span>:55:13<span class="w"> </span><span class="m">4</span>.1<span class="w"> </span>GiB<span class="w"> </span>posts.parquet
<span class="m">2023</span>-07-15<span class="w"> </span><span class="m">11</span>:16:11<span class="w"> </span><span class="m">1</span>.5<span class="w"> </span>MiB<span class="w"> </span>tags.parquet
<span class="m">2023</span>-07-15<span class="w"> </span><span class="m">11</span>:17:01<span class="w"> </span><span class="m">733</span>.7<span class="w"> </span>MiB<span class="w"> </span>users.parquet
<span class="m">2023</span>-07-15<span class="w"> </span><span class="m">11</span>:23:34<span class="w"> </span><span class="m">2</span>.2<span class="w"> </span>GiB<span class="w"> </span>votes.parquet
</code></pre></div>
<p>I'm interested in the times of the queries I'll be trying, so before
starting to run them, I turn on DuckDB's timer with:</p>
<div class="highlight"><pre><span></span><code>.timer on
</code></pre></div>
<p>Let's use the <code>users.parquet</code> dataset. You can see its schema like this:</p>
<div class="highlight"><pre><span></span><code>describe (select * from 's3://us-prd-motherduck-open-datasets/stackoverflow/parquet/2023-05/users.parquet');
┌────────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │ column_type │ null │ key │ default │ extra │
│ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │
├────────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
│ Id │ BIGINT │ YES │ │ │ │
│ Reputation │ BIGINT │ YES │ │ │ │
│ CreationDate │ TIMESTAMP │ YES │ │ │ │
│ DisplayName │ VARCHAR │ YES │ │ │ │
│ LastAccessDate │ TIMESTAMP │ YES │ │ │ │
│ AboutMe │ VARCHAR │ YES │ │ │ │
│ Views │ BIGINT │ YES │ │ │ │
│ UpVotes │ BIGINT │ YES │ │ │ │
│ DownVotes │ BIGINT │ YES │ │ │ │
└────────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┘
Run Time (s): real 0.793 user 0.015590 sys 0.007591
</code></pre></div>
<p>Running queries over S3 is going to be slow, especially from my laptop, so I copied the dataset locally. <code>users.parquet</code> is 734MB.</p>
<p>Let's run a simple query:</p>
<div class="highlight"><pre><span></span><code>D select SUM(Views) from 'users.parquet' where DisplayName like 'Petko%';
┌──────────────┐
│ sum("Views") │
│ int128 │
├──────────────┤
│ 573 │
└──────────────┘
Run Time (s): real 0.207 user 1.364285 sys 0.099568
</code></pre></div>
<p>To talk in more depth about this query, I need to mention that Parquet
files are organized in row groups. Row groups contain rows of data.
The data inside row group is stored in column chunks and Parquet keeps
track of statistics for each column chunk like min and max values.</p>
<p>This query took <code>0.207s</code>. Note that the <code>user</code> + <code>sys</code> time is more than this number.
That's because the query was running on multiple threads, which is possible
due to the data being stored in separate row groups.</p>
<p>We can explain this query too:</p>
<div class="highlight"><pre><span></span><code>D explain analyze select SUM(Views) from 'users.parquet'
where DisplayName like 'Petko%';
┌─────────────────────────────┐
│┌───────────────────────────┐│
│└───────────────────────────┘│
└─────────────────────────────┘
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││ Query Profiling Information ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
explain analyze select SUM(Views) from 'users.parquet' where DisplayName like 'Petko%';
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││ Total Time: 0.202s ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
┌───────────────────────────┐
│ EXPLAIN_ANALYZE │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 0 │
│ (0.00s) │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ UNGROUPED_AGGREGATE │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ sum(#0) │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 1 │
│ (0.00s) │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ PROJECTION │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ Views │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 119 │
│ (0.00s) │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ FILTER │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│prefix(DisplayName, 'Petko'│
│ ) │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ EC: 3988557 │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 119 │
│ (0.01s) │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ PARQUET_SCAN │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ DisplayName │
│ Views │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│Filters: DisplayName>=Petko│
│ AND DisplayName<Petkp AND│
│ DisplayName IS NOT NULL │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ EC: 3988557 │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 119 │
│ (1.57s) │
└───────────────────────────┘
Run Time (s): real 0.209 user 1.357049 sys 0.104349
</code></pre></div>
<p>Look at this section:</p>
<div class="highlight"><pre><span></span><code>┌─────────────┴─────────────┐
│ PARQUET_SCAN │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ DisplayName │
│ Views │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│Filters: DisplayName>=Petko│
│ AND DisplayName<Petkp AND│
│ DisplayName IS NOT NULL │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ EC: 3988557 │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 119 │
│ (1.57s) │
└───────────────────────────┘
</code></pre></div>
<p>To understand what's going on here and how DuckDB queries Parquet, it'll be helpful to read <a href="https://duckdb.org/2021/06/25/querying-parquet.html">Querying Parquet with Precision using DuckDB</a>.</p>
<p>What we see here is a projection and filter pushdown, as described <a href="https://duckdb.org/2021/06/25/querying-parquet.html#automatic-filter--projection-pushdown">here</a>.
DuckDB is not going to fetch the entire Parquet file, it will only fetch two
columns (projection pushdown). It can also use min/max statistics and
skip certain row groups (filter pushdown).</p>
<p>After it has applied these pushdowns, it will execute a <code>FILTER</code> and <code>PROJECTION</code>, which
happen within DuckDB.</p>
<p>This query however doesn't benefit much from the filter pushdown, since the data
is not sorted by DisplayName so the filter probably returns most of the data. The data
for <code>users.parquet</code> is sorted by Id: </p>
<div class="highlight"><pre><span></span><code>D select Id from read_parquet('users.parquet') limit 10 offset 2000;
┌───────┐
│ Id │
│ int64 │
├───────┤
│ 2808 │
│ 2811 │
│ 2812 │
│ 2813 │
│ 2815 │
│ 2818 │
│ 2819 │
│ 2820 │
│ 2821 │
│ 2822 │
└───────┘
Run Time (s): real 0.009 user 0.016330 sys 0.003842
</code></pre></div>
<p>You can also explore the Parquet Metadata to see this as well.
DuckDB offers a way to query the Parquet Metadata (<a href="https://duckdb.org/docs/data/parquet/metadata.html">link</a>).</p>
<div class="highlight"><pre><span></span><code>D select path_in_schema, row_group_id, stats_min_value, stats_max_value
from parquet_metadata('users.parquet') where (path_in_schema='Id') limit 10;
┌────────────────┬──────────────┬─────────────────┬─────────────────┐
│ path_in_schema │ row_group_id │ stats_min_value │ stats_max_value │
│ varchar │ int64 │ varchar │ varchar │
├────────────────┼──────────────┼─────────────────┼─────────────────┤
│ Id │ 0 │ -1014 │ 248850 │
│ Id │ 1 │ 248853 │ 412339 │
│ Id │ 2 │ 412342 │ 573874 │
│ Id │ 3 │ 573875 │ 735344 │
│ Id │ 4 │ 735345 │ 889114 │
│ Id │ 5 │ 889115 │ 1049570 │
│ Id │ 6 │ 1049572 │ 1211500 │
│ Id │ 7 │ 1211501 │ 1370352 │
│ Id │ 8 │ 1370353 │ 1479833 │
│ Id │ 9 │ 1479835 │ 1588348 │
├────────────────┴──────────────┴─────────────────┴─────────────────┤
│ 10 rows 4 columns │
└───────────────────────────────────────────────────────────────────┘
Run Time (s): real 0.025 user 0.023375 sys 0.000999
</code></pre></div>
<p>Unsurprisingly queries which filter by Id are going to be much faster.</p>
<p>Here's a query that returns 119 rows and filters on <code>DisplayName</code>:</p>
<div class="highlight"><pre><span></span><code>D select count(*), sum(UpVotes) from 'users.parquet' where DisplayName like 'Petko%';
┌──────────────┬──────────────┐
│ count_star() │ sum(UpVotes) │
│ int64 │ int128 │
├──────────────┼──────────────┤
│ 119 │ 330 │
└──────────────┴──────────────┘
Run Time (s): real 0.201 user 1.363518 sys 0.100677
</code></pre></div>
<p>And here's a query that returns 3110 rows and filters on Id:</p>
<div class="highlight"><pre><span></span><code>D select count(*), sum(UpVotes) from 'users.parquet' where Id >= 1000 * 1000 and Id <= 1000 * 1000 + 5000;
┌──────────────┬──────────────┐
│ count_star() │ sum(UpVotes) │
│ int64 │ int128 │
├──────────────┼──────────────┤
│ 3110 │ 235823 │
└──────────────┴──────────────┘
Run Time (s): real 0.012 user 0.014406 sys 0.003434
</code></pre></div>
<p>The latter is much faster. We can look at the <code>EXPLAIN</code> too:</p>
<div class="highlight"><pre><span></span><code>┌─────────────┴─────────────┐
│ PARQUET_SCAN │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ UpVotes │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│Filters: Id>=1000000 AND Id│
│ <=1005000 AND Id IS NOT │
│ NULL │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ EC: 3988557 │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 3110 │
│ (0.02s) │
└───────────────────────────┘
</code></pre></div>
<p>As you can see this took only 0.02s. The Parquet reader likely skipped most row groups and picked one or two.</p>
<h1 id="conclusion">Conclusion</h1>
<p>There are a few points here for me:</p>
<ul>
<li>Querying Parquet is fastest when you filter on rows which are sorted (or at least mostly sorted).</li>
<li>Parquet queries can be quite fast even if they're done directly on the Parquet data.</li>
</ul>
<h1 id="todo">TODO</h1>
<p>There are a couple of things I'm curious about exploring:</p>
<ul>
<li>Can I achieve faster execution times for some queries by splitting the data into
directories? For example, I can split by date and sort on <code>DisplayName</code>. This way queries
which filter on date and a <code>DisplayName</code> prefix could possibly be made faster.</li>
<li>If I copy the Parquet file into a DuckDB table and <a href="https://duckdb.org/docs/sql/indexes.html">create indexes</a> on it, what kind of
queries are going to become faster?</li>
</ul>In how many ways can you split a string into substrings?2017-12-04T11:53:00-08:002017-12-04T11:53:00-08:00Petko Minkovtag:pminkov.github.io,2017-12-04:/blog/in-how-many-ways-can-you-split-a-string-into-substrings.html<p>I chatted with a friend, talking about the number of ways to split a string into all possible substrings. I like how elegantly this problem can be answered by using some basic mathematics and I sat down and wrote the idea. Here's the <a href="https://github.com/pminkov/notebooks/blob/master/Number%20of%20ways%20to%20split%20a%20string%20into%20substrings.ipynb">Jupyter notebook</a> for it.</p>Simple guide to SSH2017-08-09T17:30:00-07:002017-08-09T17:30:00-07:00Petko Minkovtag:pminkov.github.io,2017-08-09:/blog/simple-guide-to-ssh.html<div class="toc"><span class="toctitle">Table of contents:</span><ul>
<li><a href="#what-is-ssh-for">What is SSH for?</a></li>
<li><a href="#the-key-pair">The key pair</a></li>
<li><a href="#how-does-ssh-work">How does SSH work?</a></li>
<li><a href="#the-ssh-command">The ssh command</a></li>
<li><a href="#the-ssh-agent-daemon">The ssh-agent daemon</a><ul>
<li><a href="#usage">Usage</a></li>
<li><a href="#starting-ssh-agent">Starting ssh-agent</a></li>
<li><a href="#ssh-agent-forwarding">ssh-agent forwarding</a></li>
</ul>
</li>
<li><a href="#tips-and-tricks">Tips and tricks</a><ul>
<li><a href="#manually-verifying-a-private-key">Manually verifying a private key</a></li>
</ul>
</li>
</ul>
</div>
<h2 id="what-is-ssh-for">What is SSH for?</h2>
<p>SSH is used to communicate securely between a client and a server. Some …</p><div class="toc"><span class="toctitle">Table of contents:</span><ul>
<li><a href="#what-is-ssh-for">What is SSH for?</a></li>
<li><a href="#the-key-pair">The key pair</a></li>
<li><a href="#how-does-ssh-work">How does SSH work?</a></li>
<li><a href="#the-ssh-command">The ssh command</a></li>
<li><a href="#the-ssh-agent-daemon">The ssh-agent daemon</a><ul>
<li><a href="#usage">Usage</a></li>
<li><a href="#starting-ssh-agent">Starting ssh-agent</a></li>
<li><a href="#ssh-agent-forwarding">ssh-agent forwarding</a></li>
</ul>
</li>
<li><a href="#tips-and-tricks">Tips and tricks</a><ul>
<li><a href="#manually-verifying-a-private-key">Manually verifying a private key</a></li>
</ul>
</li>
</ul>
</div>
<h2 id="what-is-ssh-for">What is SSH for?</h2>
<p>SSH is used to communicate securely between a client and a server. Some examples that most people are familiar with are SSH-ing to a remote server or using GitHub.</p>
<h2 id="the-key-pair">The key pair</h2>
<p>To establish an SSH connection you need a <strong>private key</strong> and a <strong>public key</strong>. You keep the private key in a safe place (usually in <code>~/.ssh/</code>) and upload the public key to any server that you want to securely connect to. Encryption with a private key and a public key is called <a href="https://en.wikipedia.org/wiki/Public-key_cryptography">assymetric encryption</a>.</p>
<p>How do you create a key pair. Like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ssh-keygen<span class="w"> </span>-t<span class="w"> </span>rsa<span class="w"> </span>-b<span class="w"> </span><span class="m">4096</span><span class="w"> </span>-C<span class="w"> </span><span class="s2">"myemail@gmail.com"</span>
Generating<span class="w"> </span>public/private<span class="w"> </span>rsa<span class="w"> </span>key<span class="w"> </span>pair.
Enter<span class="w"> </span>file<span class="w"> </span><span class="k">in</span><span class="w"> </span>which<span class="w"> </span>to<span class="w"> </span>save<span class="w"> </span>the<span class="w"> </span>key<span class="w"> </span><span class="o">(</span>/home/petko/.ssh/id_rsa<span class="o">)</span>:<span class="w"> </span>./mykey
Enter<span class="w"> </span>passphrase<span class="w"> </span><span class="o">(</span>empty<span class="w"> </span><span class="k">for</span><span class="w"> </span>no<span class="w"> </span>passphrase<span class="o">)</span>:<span class="w"> </span>
Enter<span class="w"> </span>same<span class="w"> </span>passphrase<span class="w"> </span>again:<span class="w"> </span>
Your<span class="w"> </span>identification<span class="w"> </span>has<span class="w"> </span>been<span class="w"> </span>saved<span class="w"> </span><span class="k">in</span><span class="w"> </span>./mykey.
Your<span class="w"> </span>public<span class="w"> </span>key<span class="w"> </span>has<span class="w"> </span>been<span class="w"> </span>saved<span class="w"> </span><span class="k">in</span><span class="w"> </span>./mykey.pub.
The<span class="w"> </span>key<span class="w"> </span>fingerprint<span class="w"> </span>is:
SHA256:0/zlEZZj0R6yM4uHVD+xJWoLOx3+TormCRcyWt3c53o<span class="w"> </span>myemail@gmail.com
The<span class="w"> </span>key<span class="err">'</span>s<span class="w"> </span>randomart<span class="w"> </span>image<span class="w"> </span>is:
+---<span class="o">[</span>RSA<span class="w"> </span><span class="m">4096</span><span class="o">]</span>----+
<span class="p">|</span><span class="w"> </span>..<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>o.<span class="o">=</span>o<span class="p">|</span>
<span class="p">|</span><span class="w"> </span>..Xo<span class="o">=</span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>+.++B<span class="w"> </span>*.<span class="p">|</span>
<span class="p">|</span><span class="w"> </span>S<span class="w"> </span>**<span class="o">=</span>oB<span class="w"> </span>o<span class="p">|</span>
<span class="p">|</span><span class="w"> </span>o<span class="w"> </span>+o<span class="o">=</span>+<span class="o">=</span><span class="w"> </span>+<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>.<span class="w"> </span>.<span class="w"> </span>..o.o<span class="w"> </span>.<span class="p">|</span>
<span class="p">|</span><span class="w"> </span>o.o<span class="w"> </span>o..E<span class="p">|</span>
<span class="p">|</span><span class="w"> </span>o+<span class="w"> </span>..o.<span class="w"> </span><span class="p">|</span>
+----<span class="o">[</span>SHA256<span class="o">]</span>-----+
$<span class="w"> </span>ls<span class="w"> </span>-l
total<span class="w"> </span><span class="m">8</span>
-rw-------<span class="w"> </span><span class="m">1</span><span class="w"> </span>petko<span class="w"> </span>petko<span class="w"> </span><span class="m">3243</span><span class="w"> </span>Aug<span class="w"> </span><span class="m">9</span><span class="w"> </span><span class="m">16</span>:24<span class="w"> </span>mykey
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>petko<span class="w"> </span>petko<span class="w"> </span><span class="m">743</span><span class="w"> </span>Aug<span class="w"> </span><span class="m">9</span><span class="w"> </span><span class="m">16</span>:24<span class="w"> </span>mykey.pub
</code></pre></div>
<p>How do the public and private key look like. They look like this (I have removed some of the output to keep you from scrolling too much):</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>./mykey
-----BEGIN<span class="w"> </span>RSA<span class="w"> </span>PRIVATE<span class="w"> </span>KEY-----
MIIJKQIBAAKCAgEA2FGlZybznkcHQG530bj+DlrY74nTh+shP1uyJA25BrkAyOz9
xc8Tvlk3QfBcGFfQKc1OowV80XtNyXXnOeFTqlh8B8DS1Mul165wgb+pJDROvI0J
...
3vZfDPXo9w2XwAwN7hLimCVWdqr0JI8BmbussW4ZrJRcra1rvsLj6sip9Ry3oP+9
PIUEPwDY/YUVRZV2De4cBdBnwTmj9RoXOW63mW6sL8lfeYjJQwQys+jVVjRi
-----END<span class="w"> </span>RSA<span class="w"> </span>PRIVATE<span class="w"> </span>KEY-----
$<span class="w"> </span>cat<span class="w"> </span>./mykey.pub<span class="w"> </span>
ssh-rsa<span class="w"> </span>AAAAB3NzaC<span class="w"> </span>...<span class="w"> </span>+PAAKfQ<span class="o">==</span><span class="w"> </span>myemail@gmail.com
</code></pre></div>
<p>So that's it. A private key and a public key. Nothing else. Keep the private key secret, upload the public key.</p>
<h2 id="how-does-ssh-work">How does SSH work?</h2>
<p>Great question. If you have a key pair you can encrypt a message with one of the keys and decrypt it with the other key. But that's not how data is exchanged with an SSH connection. SSH is actually using <a href="https://en.wikipedia.org/wiki/Symmetric-key_algorithm">symmetric key encryption</a>. The private and public keys are used to securely exchange a temporary symmetric key used to encrypt the data between two machines. The symmetric key crosses the wire in encrypted form, so an attacked can't find out what the key is.</p>
<p>Dig into this excellent <a href="https://superuser.com/questions/383732/how-does-ssh-encryption-work">stackoverflow question</a> to learn more about this.</p>
<h2 id="the-ssh-command">The ssh command</h2>
<p>We now know what's involved in SSH communication. Let's look at the commands that are used. </p>
<p>We'll start with a clean example. I have an AWS EC2 instance on which I have installed a public key. The private key is located in my <code>~/.ssh/</code> directory. It's called <code>aws-laptop</code>, because I use it from my laptop.</p>
<p>Let's try to ssh into the instance:</p>
<div class="highlight"><pre><span></span><code>~/.ssh$<span class="w"> </span>ssh<span class="w"> </span>ubuntu@35.165.19.203
Permission<span class="w"> </span>denied<span class="w"> </span><span class="o">(</span>publickey<span class="o">)</span>.
</code></pre></div>
<p>Doesn't work. Well, of course - ssh doesn't know what private key to use. We have to point to it using the <code>-i</code> option.</p>
<div class="highlight"><pre><span></span><code>~/.ssh$<span class="w"> </span>ssh<span class="w"> </span>-i<span class="w"> </span>./aws-laptop<span class="w"> </span>ubuntu@35.165.19.203
The<span class="w"> </span>authenticity<span class="w"> </span>of<span class="w"> </span>host<span class="w"> </span><span class="s1">'35.165.19.203 (35.165.19.203)'</span><span class="w"> </span>can<span class="s1">'t be established.</span>
<span class="s1">ECDSA key fingerprint is SHA256: 3w+LlpD/HH .......</span>
<span class="s1">Are you sure you want to continue connecting (yes/no)? yes</span>
<span class="s1">Enter passphrase for key '</span>./aws-laptop<span class="err">'</span>:<span class="w"> </span><span class="o">[</span>Here<span class="w"> </span>I<span class="w"> </span>entered<span class="w"> </span>a<span class="w"> </span>passphrase<span class="o">]</span>
Welcome<span class="w"> </span>to<span class="w"> </span>Ubuntu<span class="w"> </span><span class="m">16</span>.04.2<span class="w"> </span>LTS<span class="w"> </span><span class="o">(</span>GNU/Linux<span class="w"> </span><span class="m">4</span>.4.0-1022-aws<span class="w"> </span>x86_64<span class="o">)</span>
...
Last<span class="w"> </span>login:<span class="w"> </span>Wed<span class="w"> </span>Aug<span class="w"> </span><span class="m">9</span><span class="w"> </span><span class="m">23</span>:09:13<span class="w"> </span><span class="m">2017</span><span class="w"> </span>from<span class="w"> </span><span class="m">76</span>.102.141.14
ubuntu@ip-172-31-13-46:~$
</code></pre></div>
<p>Hooray, I'm in! What happened in the meantime is that the ssh command saved an entry into the <code>known_hosts</code> file, so next time I ssh it won't ask me again to confirm the authenticity of the remote host.</p>
<p>However, I'll still need to point to the private key and enter the passphrase for it. Very inconvenient.</p>
<h2 id="the-ssh-agent-daemon">The ssh-agent daemon</h2>
<p>The solution to the above problem is the <code>ssh-agent</code> daemon. <code>ssh-agent</code> is a daemon that you start and you can add private keys to it so that next time you ssh into a server, you don't have to point to them.</p>
<h3 id="usage">Usage</h3>
<p>How does ssh-agent know which private key to use? When you run <code>ssh -v ubuntu@35.165.19.203</code> you can see that there's a bit of back and forth where <code>ssh</code> seems to be trying out private keys from <code>ssh-agent</code>. So I guess that's how it's done:</p>
<div class="highlight"><pre><span></span><code>debug1:<span class="w"> </span>Offering<span class="w"> </span>RSA<span class="w"> </span>public<span class="w"> </span>key:<span class="w"> </span>/home/petko/.ssh/id_rsa
debug1:<span class="w"> </span>Authentications<span class="w"> </span>that<span class="w"> </span>can<span class="w"> </span><span class="k">continue</span>:<span class="w"> </span>publickey
debug1:<span class="w"> </span>Offering<span class="w"> </span>RSA<span class="w"> </span>public<span class="w"> </span>key:<span class="w"> </span>./aws-laptop
debug1:<span class="w"> </span>Server<span class="w"> </span>accepts<span class="w"> </span>key:<span class="w"> </span>pkalg<span class="w"> </span>rsa-sha2-512<span class="w"> </span>blen<span class="w"> </span><span class="m">279</span>
</code></pre></div>
<p>We can list the keys in ssh-agent with this command:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ssh-add<span class="w"> </span>-l
The<span class="w"> </span>agent<span class="w"> </span>has<span class="w"> </span>no<span class="w"> </span>identities.
</code></pre></div>
<p>No keys. So let's add the key that we were using.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ssh-add<span class="w"> </span>./aws-laptop
Enter<span class="w"> </span>passphrase<span class="w"> </span><span class="k">for</span><span class="w"> </span>./aws-laptop:<span class="w"> </span><span class="o">[</span>I<span class="w"> </span>entered<span class="w"> </span>passphrase<span class="o">]</span>
Identity<span class="w"> </span>added:<span class="w"> </span>./aws-laptop<span class="w"> </span><span class="o">(</span>./aws-laptop<span class="o">)</span>
$<span class="w"> </span>ssh-add<span class="w"> </span>-l
<span class="m">2048</span><span class="w"> </span>SHA256:tU++v7cfD8...<span class="w"> </span>./aws-laptop<span class="w"> </span><span class="o">(</span>RSA<span class="o">)</span>
</code></pre></div>
<p>The private key is now saved! I can now ssh without pointing to it:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ssh<span class="w"> </span>ubuntu@35.165.19.203
Welcome<span class="w"> </span>to<span class="w"> </span>Ubuntu<span class="w"> </span><span class="m">16</span>.04.2<span class="w"> </span>LTS<span class="w"> </span><span class="o">(</span>GNU/Linux<span class="w"> </span><span class="m">4</span>.4.0-1022-aws<span class="w"> </span>x86_64<span class="o">)</span>
...
ubuntu@ip-172-31-13-46:~$<span class="w"> </span>
</code></pre></div>
<h3 id="starting-ssh-agent">Starting ssh-agent</h3>
<p>You don't want to start ssh-agent manually. I'll describe my setup for automatically staring it in Ubuntu. In my <code>~/.bash_profile</code> I have the following:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Start ssh-agent.</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-S<span class="w"> </span>~/.ssh/ssh_auth_sock<span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Starting ssh-agent"</span>
<span class="w"> </span><span class="nb">eval</span><span class="w"> </span><span class="sb">`</span>ssh-agent<span class="sb">`</span>
<span class="w"> </span>ln<span class="w"> </span>-sf<span class="w"> </span><span class="s2">"</span><span class="nv">$SSH_AUTH_SOCK</span><span class="s2">"</span><span class="w"> </span>~/.ssh/ssh_auth_sock
<span class="k">fi</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">SSH_AUTH_SOCK</span><span class="o">=</span>~/.ssh/ssh_auth_sock
ssh-add<span class="w"> </span>-l<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">"The agent has no identities"</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span>ssh-add
</code></pre></div>
<p>Remember that <code>.bash_profile</code> is executed for interactive login shells. An interactive login shell is what's run when you ssh into a machine. One interesting thing about Ubuntu is that if you open a shell from within Gnome, it's an interactive shell, but it's not a login shell. Gnome also runs a separate ssh-agent, which you can find out when you see the contents of the <code>SSH_AUTH_SOCK</code> environment variable. But I don't use shells from within Gnome, so that's not important to me. If I had a desktop Ubuntu, I'd care about that situation more.</p>
<p>Here's an <a href="http://www.snailbook.com/faq/about-agent.auto.html">article</a> that's a good description of the mechanics of starting an ssh-agent.</p>
<h3 id="ssh-agent-forwarding">ssh-agent forwarding</h3>
<p>So you've setup SSH on your local machine, but you want to SSH to your cloud server and pull some git repo. Or you want to SSH from the cloud server to another server, or maybe copy some files with <code>scp</code>. You can generate a key pair for the cloud server and distribute the public key to all machines and services that you want that cloud server to have access to. But that's not convenient. This is where SSH agent forwarding is needed. With SSH agent forwarding, you ssh into a machine and it takes the private keys loaded in your ssh-agent with you to the other machine.</p>
<p>GitHub has a <a href="https://developer.github.com/v3/guides/using-ssh-agent-forwarding/">great article</a> on how to setup that. One thing that I'd add to it is that you don't have to start ssh-agent on the target machine! It's automatically started when you ssh to it.</p>
<p>In my particular setup, I'm ssh-ing to a machine that has just an IP address, so I added an entry for it in <code>/etc/hosts</code>:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>/etc/hosts
<span class="m">127</span>.0.0.1<span class="w"> </span>localhost
<span class="m">52</span>.39.10.196<span class="w"> </span>webhost
...
</code></pre></div>
<p>My <code>~/.ssh/config</code> file looks like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>./config<span class="w"> </span>
Host<span class="w"> </span>webhost
<span class="w"> </span>ForwardAgent<span class="w"> </span>yes
<span class="w"> </span>User<span class="w"> </span>ubuntu
</code></pre></div>
<p>So now to ssh to this machine, I only type <code>ssh webhost</code>. Once I ssh there I can run <code>ssh-add -l</code> and see that my keys are available on the remote machine.</p>
<h2 id="tips-and-tricks">Tips and tricks</h2>
<h3 id="manually-verifying-a-private-key">Manually verifying a private key</h3>
<p>Today I was trying again and again to connect to a remote machine using the wrong private key. I was able to connect to that machine from my laptop, but not from my Ubuntu VM.</p>
<p>I was running the following command:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ssh<span class="w"> </span>-i<span class="w"> </span>./somekey.pem<span class="w"> </span>ubuntu@35.165.19.203
Permission<span class="w"> </span>denied<span class="w"> </span><span class="o">(</span>publickey<span class="o">)</span>.
</code></pre></div>
<p>How did I debug this? I ssh-ed into the server from my laptop and looked up the public key in <code>~/.ssh/authorized_keys</code>. I then generated the public key from my private key, like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ssh-keygen<span class="w"> </span>-y<span class="w"> </span>-f<span class="w"> </span>./somekey.pem
ssh-rsa<span class="w"> </span>AAAAB3....KY1
</code></pre></div>
<p>And sure enough, the public keys were not matching. I then found the correct private key and was able to connect with it.</p>
<p>If you're following carefully, you'll notice that I did something here that's considered a bad practice. I reused my private key from my laptop on my VirtualBox. I should generate a new key pair and add the public key to the <code>authorized_keys</code> file on the server.</p>
<p>So that's it so far. Enjoy your remote access.</p>Socket Programming in Linux2017-08-02T12:10:00-07:002017-08-02T12:10:00-07:00Petko Minkovtag:pminkov.github.io,2017-08-02:/blog/socket-programming-in-linux.html<p>Some time last year I wanted to learn more about network programming and multi threading and wrote a <a href="https://github.com/pminkov/webserver/">webserver</a> in C. A few days ago I decided that my learning in network programming is not quite complete without writing a client as well, so I wrote <a href="https://github.com/pminkov/webserver/blob/master/client.c">a client</a> too. It's …</p><p>Some time last year I wanted to learn more about network programming and multi threading and wrote a <a href="https://github.com/pminkov/webserver/">webserver</a> in C. A few days ago I decided that my learning in network programming is not quite complete without writing a client as well, so I wrote <a href="https://github.com/pminkov/webserver/blob/master/client.c">a client</a> too. It's now time to write a bit about how all of this works.</p>
<p>Programming sockets in Linux is similar to all system programming in C. It's easy to make errors, so you have to be careful. It's difficult to comprehend the complexity and intricacy if you're just copying and pasting code from Stack Overflow. <a href="http://man7.org/tlpi/">The Linux Programming Interface</a> is a book that has several chapters dedicated to socket programming and it describes it very well, as well as throwing in a lot of information about how networks work and an excellent concise description of the TCP protocol. I highly recommend that book and I might write more about it in the future.</p>
<p>So let's talk a bit about how to write a webserver and a client for it in C.</p>
<p>Let's start with the server. The full code is in <a href="https://github.com/pminkov/webserver/blob/master/server.c">server.c</a>, but I'll summarize the imporatnt points:</p>
<ul>
<li>Create a socket with <code>socket()</code>. The code looks like this:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="kt">int</span><span class="w"> </span><span class="n">sockfd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">socket</span><span class="p">(</span><span class="n">AF_INET</span><span class="p">,</span><span class="w"> </span><span class="n">SOCK_STREAM</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span>
</code></pre></div>
<p><code>SOCK_STREAM</code> means that we're creating a stream socket (TCP socket).</p>
<ul>
<li>Bind the socket to an address. The key lines are:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="mi">1</span><span class="o">:</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">sockaddr_in</span><span class="w"> </span><span class="n">serv_addr</span><span class="p">;</span>
<span class="mi">2</span><span class="o">:</span><span class="w"> </span><span class="kt">uint16_t</span><span class="w"> </span><span class="n">port</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">8000</span><span class="p">;</span>
<span class="p">...</span>
<span class="mi">3</span><span class="o">:</span><span class="w"> </span><span class="n">serv_addr</span><span class="p">.</span><span class="n">sin_family</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">AF_INET</span><span class="p">;</span>
<span class="mi">4</span><span class="o">:</span><span class="w"> </span><span class="n">serv_addr</span><span class="p">.</span><span class="n">sin_port</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">htons</span><span class="p">(</span><span class="n">port</span><span class="p">);</span>
<span class="mi">5</span><span class="o">:</span><span class="w"> </span><span class="n">serv_addr</span><span class="p">.</span><span class="n">sin_addr</span><span class="p">.</span><span class="n">s_addr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">htonl</span><span class="p">(</span><span class="n">INADDR_ANY</span><span class="p">);</span>
<span class="p">...</span>
<span class="mi">6</span><span class="o">:</span><span class="w"> </span><span class="n">bind</span><span class="p">(</span><span class="n">sockfd</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">sockaddr</span><span class="w"> </span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="o">&</span><span class="n">serv_addr</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">serv_addr</span><span class="p">))</span>
</code></pre></div>
<p>We're creating an address <code>serv_addr</code> and binding it to the socket we created above. Note that we use <code>INADDR_ANY</code> as the IP address on which the socket is listening. This is the so-called IPv4 wildcard address. It binds your server to all interfaces available on your machine. That's more useful for me than binding to <code>INADDR_LOOPBACK</code> (127.0.0.1), because I'm running my code in a VM and accessing the server from my MacBook. Run <code>ifconfig</code> on your box to see the network interfaces that are available on it.</p>
<ul>
<li>Great, we have a socket and it's assigned to an address. This socket can now be used as as passive socket (it's listening for connections) or an active socket (it's used to connect to a peer socket). We want a passive socket. We mark it as a passive socket by calling the <code>listen</code> function. The code looks like this:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">listen</span><span class="p">(</span><span class="n">sockfd</span><span class="p">,</span><span class="w"> </span><span class="n">SOMAXCONN</span><span class="p">)</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="n">error</span><span class="p">(</span><span class="s">"Couldn't listen"</span><span class="p">);</span>
</code></pre></div>
<ul>
<li>The socket is now ready to accept connections. This can happen in a loop that looks like this:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="k">struct</span><span class="w"> </span><span class="nc">sockaddr_in</span><span class="w"> </span><span class="n">client_addr</span><span class="p">;</span>
<span class="kt">int</span><span class="w"> </span><span class="n">cli_len</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">client_addr</span><span class="p">);</span>
<span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">newsockfd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">accept</span><span class="p">(</span><span class="n">sockfd</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">sockaddr</span><span class="w"> </span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="o">&</span><span class="n">client_addr</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="kt">socklen_t</span><span class="w"> </span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="o">&</span><span class="n">cli_len</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">newsockfd</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="n">error</span><span class="p">(</span><span class="s">"Error on accept"</span><span class="p">);</span>
<span class="w"> </span><span class="c1">// newsockfd is a file descriptor. </span>
<span class="w"> </span><span class="c1">// Handle the connection by reading from and writing to this descriptor.</span>
<span class="w"> </span><span class="p">...</span>
<span class="p">}</span>
</code></pre></div>
<p>The <code>accept</code> function blocks until a connection request is received and creates a new socket when that happens. After that, it's your call how you'll handle this socket. You can fork a new process, create a new thread or in my case I have a thred pool of pre-forked threads which just pull these sockets from a queue and do what's necessary to handle a web server request.</p>
<p>That's the server. Now let's look at the client. Full code is in <a href="https://github.com/pminkov/webserver/blob/master/client.c">client.c</a>. It's a pretty simple client that connects to localhost (127.0.0.1). So you can't use it if the server is running remotely.</p>
<p>You start by creating a socket with <code>socket()</code> and then calling <code>connect()</code> and passing the address to which you want to connect:</p>
<div class="highlight"><pre><span></span><code><span class="kt">int</span><span class="w"> </span><span class="n">sockfd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">socket</span><span class="p">(</span><span class="n">AF_INET</span><span class="p">,</span><span class="w"> </span><span class="n">SOCK_STREAM</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">sockfd</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">-1</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">error</span><span class="p">(</span><span class="s">"socket"</span><span class="p">);</span>
<span class="p">}</span>
<span class="kt">uint16_t</span><span class="w"> </span><span class="n">port</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">8000</span><span class="p">;</span>
<span class="k">struct</span><span class="w"> </span><span class="nc">sockaddr_in</span><span class="w"> </span><span class="n">serv_addr</span><span class="p">;</span>
<span class="n">serv_addr</span><span class="p">.</span><span class="n">sin_family</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">AF_INET</span><span class="p">;</span>
<span class="n">serv_addr</span><span class="p">.</span><span class="n">sin_port</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">htons</span><span class="p">(</span><span class="n">port</span><span class="p">);</span>
<span class="n">serv_addr</span><span class="p">.</span><span class="n">sin_addr</span><span class="p">.</span><span class="n">s_addr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">htonl</span><span class="p">(</span><span class="n">INADDR_LOOPBACK</span><span class="p">);</span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">connect</span><span class="p">(</span><span class="n">sockfd</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">sockaddr</span><span class="w"> </span><span class="o">*</span><span class="p">)</span><span class="o">&</span><span class="n">serv_addr</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">sockaddr_in</span><span class="p">))</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">-1</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">error</span><span class="p">(</span><span class="s">"connect"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div>
<p>If the connection is successful, you can just send your request to the server using the <code>write()</code> system call and read the response with the <code>read()</code> call.</p>
<p>So that's a simple web server and a client. The code still has a lot of bugs (the client will buffer overflow if I pass a request string longer than 512 chars, I'm not closing the connection explicitly, etc.), but it gets the job done.</p>
<p>A final interesting thing to do is to run Wireshark and see what packets are going between the client and the server. To do this, I started the server on my Virtual box Ubuntu (192.168.1.135) and ran <code>curl</code> from my MacBook (192.168.1.86). Here's what I saw in Wireshark:</p>
<p><center><img alt="Wireshark" src="https://pminkov.github.io/blog/images/wireshark.png"></center></p>
<p>You can see the three way TCP handshake in the beginning (<code>SYN</code> followed by <code>SYN,ACK</code> followed by <code>ACK</code>) and everything else in between. It's definitely something interesting to try out.</p>Versioning Docker Images2017-07-30T21:57:00-07:002017-07-30T21:57:00-07:00Petko Minkovtag:pminkov.github.io,2017-07-30:/blog/versioning-docker-images.html<p>If you're running Docker containers in the cloud, you're probably uploading them into a registry. If you're using Google Cloud, that would be <a href="https://cloud.google.com/container-registry/">gcr.io </a>(Google Container Registry).</p>
<p>As you're iterating on your application, you'll need to push new Docker images to the registry. A natural questions that comes is …</p><p>If you're running Docker containers in the cloud, you're probably uploading them into a registry. If you're using Google Cloud, that would be <a href="https://cloud.google.com/container-registry/">gcr.io </a>(Google Container Registry).</p>
<p>As you're iterating on your application, you'll need to push new Docker images to the registry. A natural questions that comes is how to version these images? You don't want to overwrite images using the same tag and it's cumbersome to keep track of increasing version numbers. A good versioning scheme is to use a <a href="https://blog.thoughtram.io/git/2014/11/18/the-anatomy-of-a-git-commit.html">git commit hash</a>. So your image name might looks like this: <code>gcr.io/kubeproject-172120/simple:88d38d9</code>. If you take your git repository at this hash, you'll find the files that produced this exact image.</p>
<p>This sounds simple to implement. You get the last commit's hash, build the image using the hash as a tag and push it to the registry. There's one big inconvinience to this scheme though - you have to commit each change if you want to use a new hash (and you do, you don't want to overwrite your production image) and when you're iterating on an image, that gets tiresome quickly. One possible solution would be to commit "debug" images. These images might be tagged with something like this <code>88d38d9-debug</code>. This is an image produced by taking the git repo at the <code>88d38d9</code> hash and making some modifications. You'll know not to include these images in your production files and it's ok to overwrite them as you're iterating.</p>
<p>So let's look at how all of this can be implemented. Let's say you're putting all your images in one directory. The contents of this directory might look like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tree
.
├──<span class="w"> </span>build-image.sh
└──<span class="w"> </span>simple
<span class="w"> </span>├──<span class="w"> </span>app.py
<span class="w"> </span>├──<span class="w"> </span>Dockerfile
<span class="w"> </span>└──<span class="w"> </span>requirements.txt
<span class="m">1</span><span class="w"> </span>directory,<span class="w"> </span><span class="m">4</span><span class="w"> </span>files
</code></pre></div>
<p>The <code>build-image.sh</code> script builds the Docker image and pushes it to <code>gcr.io</code>.</p>
<p>The script itself looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-z<span class="w"> </span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Usage: </span><span class="nv">$0</span><span class="s2"> <image_dir> [--debug]"</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span>
<span class="k">fi</span>
<span class="nv">IMAGE_NAME</span><span class="o">=</span><span class="nv">$1</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="nv">$2</span><span class="s2">"</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">"--debug"</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="c1"># If we're debugging, we can push code that's not committed.</span>
<span class="w"> </span><span class="nv">APPEND</span><span class="o">=</span><span class="s2">"-debug"</span>
<span class="k">else</span>
<span class="w"> </span><span class="nv">IMAGE_PATH</span><span class="o">=</span>/<span class="nv">$IMAGE_NAME</span>/
<span class="w"> </span><span class="k">if</span><span class="w"> </span>git<span class="w"> </span>status<span class="w"> </span>.<span class="w"> </span>--porcelain<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="nv">$IMAGE_PATH</span><span class="w"> </span>><span class="w"> </span>/dev/null<span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"You have uncommited changes to your Docker image. Please commit them"</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"before building and populating. This helps ensure that all docker images"</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"are traceable back to a git commit."</span>
<span class="w"> </span><span class="nb">echo</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Or if you're just building a debug image, use the --debug flag."</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="k">fi</span>
<span class="k">fi</span>
<span class="c1"># Set image tag.</span>
<span class="nv">GIT_REV</span><span class="o">=</span><span class="k">$(</span>git<span class="w"> </span>log<span class="w"> </span>-n<span class="w"> </span><span class="m">1</span><span class="w"> </span>--pretty<span class="o">=</span>format:%h<span class="w"> </span>--<span class="w"> </span>./<span class="nv">$IMAGE_NAME</span>/<span class="k">)</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span><span class="nv">$GIT_REV</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"You're trying to build an image that has never been committed."</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s2">"You need to commit at least one version."</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span>
<span class="k">fi</span>
<span class="nv">TAG</span><span class="o">=</span><span class="s2">"</span><span class="nv">$GIT_REV</span><span class="s2">""</span><span class="nv">$APPEND</span><span class="s2">"</span>
<span class="c1"># Set image repo.</span>
<span class="nv">PROJECT_ID</span><span class="o">=</span><span class="k">$(</span>gcloud<span class="w"> </span>config<span class="w"> </span>get-value<span class="w"> </span>project<span class="w"> </span><span class="m">2</span>>/dev/null<span class="k">)</span>
<span class="nv">DOCKER_REPO</span><span class="o">=</span><span class="s2">"gcr.io/</span><span class="nv">$PROJECT_ID</span><span class="s2">"</span>
<span class="c1"># Full image name.</span>
<span class="nv">IMAGE_SPEC</span><span class="o">=</span><span class="s2">"</span><span class="nv">$DOCKER_REPO</span><span class="s2">/</span><span class="nv">$IMAGE_NAME</span><span class="s2">:</span><span class="nv">$TAG</span><span class="s2">"</span>
<span class="nb">cd</span><span class="w"> </span><span class="nv">$IMAGE_NAME</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-f<span class="w"> </span><span class="nv">$DOCKERFILE</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"No such file: </span><span class="nv">$IMAGE_NAME</span><span class="s2">/</span><span class="nv">$DOCKERFILE</span><span class="s2">"</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span>
<span class="k">fi</span>
<span class="nb">echo</span><span class="w"> </span><span class="nv">$IMAGE_SPEC</span>
docker<span class="w"> </span>build<span class="w"> </span>-t<span class="w"> </span><span class="nv">$IMAGE_SPEC</span><span class="w"> </span>.
gcloud<span class="w"> </span>docker<span class="w"> </span>--<span class="w"> </span>push<span class="w"> </span><span class="nv">$IMAGE_SPEC</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"Pushed </span><span class="nv">$IMAGE_SPEC</span><span class="s2">"</span>
</code></pre></div>
<p>One thing to pay attention to is that the hash that we're using is the hash of the last commit to the directory that contains the container files. This way, if you want to push a production ready image (non-debug), you can only commit the files inside this directory and if you're still working on others outside of it, you can continue doing so.</p>
<p>Let's run the <code>build-image.sh</code> script:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>./build-image.sh<span class="w"> </span>simple
gcr.io/kubehub-172120/simple:12430ce
Sending<span class="w"> </span>build<span class="w"> </span>context<span class="w"> </span>to<span class="w"> </span>Docker<span class="w"> </span>daemon<span class="w"> </span><span class="m">4</span>.096<span class="w"> </span>kB
Step<span class="w"> </span><span class="m">1</span>/9<span class="w"> </span>:<span class="w"> </span>FROM<span class="w"> </span>ubuntu:latest
<span class="w"> </span>---><span class="w"> </span>14f60031763d
Step<span class="w"> </span><span class="m">2</span>/9<span class="w"> </span>:<span class="w"> </span>MAINTAINER<span class="w"> </span>Petko<span class="w"> </span>Minkov<span class="w"> </span><span class="s2">"pminkov@gmail.com"</span>
<span class="w"> </span>---><span class="w"> </span>Using<span class="w"> </span>cache
<span class="w"> </span>---><span class="w"> </span>5a371036a9e3
Step<span class="w"> </span><span class="m">3</span>/9<span class="w"> </span>:<span class="w"> </span>RUN<span class="w"> </span>apt-get<span class="w"> </span>update<span class="w"> </span>-y
<span class="w"> </span>---><span class="w"> </span>Using<span class="w"> </span>cache
<span class="w"> </span>---><span class="w"> </span>8992277faa20
Step<span class="w"> </span><span class="m">4</span>/9<span class="w"> </span>:<span class="w"> </span>RUN<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>python-pip<span class="w"> </span>python-dev<span class="w"> </span>build-essential
<span class="w"> </span>---><span class="w"> </span>Using<span class="w"> </span>cache
<span class="w"> </span>---><span class="w"> </span>9c0937facaf0
Step<span class="w"> </span><span class="m">5</span>/9<span class="w"> </span>:<span class="w"> </span>COPY<span class="w"> </span>.<span class="w"> </span>/app
<span class="w"> </span>---><span class="w"> </span>Using<span class="w"> </span>cache
<span class="w"> </span>---><span class="w"> </span>dd9f289c1f55
Step<span class="w"> </span><span class="m">6</span>/9<span class="w"> </span>:<span class="w"> </span>WORKDIR<span class="w"> </span>/app
<span class="w"> </span>---><span class="w"> </span>Using<span class="w"> </span>cache
<span class="w"> </span>---><span class="w"> </span>d93c62ac371a
Step<span class="w"> </span><span class="m">7</span>/9<span class="w"> </span>:<span class="w"> </span>RUN<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>--upgrade<span class="w"> </span>pip
<span class="w"> </span>---><span class="w"> </span>Using<span class="w"> </span>cache
<span class="w"> </span>---><span class="w"> </span>cb2f0a65c93f
Step<span class="w"> </span><span class="m">8</span>/9<span class="w"> </span>:<span class="w"> </span>RUN<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>requirements.txt
<span class="w"> </span>---><span class="w"> </span>Using<span class="w"> </span>cache
<span class="w"> </span>---><span class="w"> </span>d8fd659127d9
Step<span class="w"> </span><span class="m">9</span>/9<span class="w"> </span>:<span class="w"> </span>CMD<span class="w"> </span>python<span class="w"> </span>app.py
<span class="w"> </span>---><span class="w"> </span>Using<span class="w"> </span>cache
<span class="w"> </span>---><span class="w"> </span>8493c8ad1a01
Successfully<span class="w"> </span>built<span class="w"> </span>8493c8ad1a01
The<span class="w"> </span>push<span class="w"> </span>refers<span class="w"> </span>to<span class="w"> </span>a<span class="w"> </span>repository<span class="w"> </span><span class="o">[</span>gcr.io/kubehub-172120/simple<span class="o">]</span>
dacb974e8350:<span class="w"> </span>Layer<span class="w"> </span>already<span class="w"> </span>exists<span class="w"> </span>
6c4d57527510:<span class="w"> </span>Layer<span class="w"> </span>already<span class="w"> </span>exists<span class="w"> </span>
5348dff0fc19:<span class="w"> </span>Layer<span class="w"> </span>already<span class="w"> </span>exists<span class="w"> </span>
738da70fc9f8:<span class="w"> </span>Layer<span class="w"> </span>already<span class="w"> </span>exists<span class="w"> </span>
f665434eb0ee:<span class="w"> </span>Layer<span class="w"> </span>already<span class="w"> </span>exists<span class="w"> </span>
26b126eb8632:<span class="w"> </span>Layer<span class="w"> </span>already<span class="w"> </span>exists<span class="w"> </span>
220d34b5f6c9:<span class="w"> </span>Layer<span class="w"> </span>already<span class="w"> </span>exists<span class="w"> </span>
8a5132998025:<span class="w"> </span>Layer<span class="w"> </span>already<span class="w"> </span>exists<span class="w"> </span>
aca233ed29c3:<span class="w"> </span>Layer<span class="w"> </span>already<span class="w"> </span>exists<span class="w"> </span>
e5d2f035d7a4:<span class="w"> </span>Layer<span class="w"> </span>already<span class="w"> </span>exists<span class="w"> </span>
12430ce:<span class="w"> </span>digest:<span class="w"> </span>sha256:51cd80db604d1ffa5230289c1f3fe40d19b3b8dc2afb0a0c003713360b07d2c6<span class="w"> </span>size:<span class="w"> </span><span class="m">2411</span>
Pushed<span class="w"> </span>gcr.io/kubehub-172120/simple:12430ce
</code></pre></div>
<p>Great, now the image is pushed. But I always like to use a "trust but verify" policy, so let's see how can we dig into what's going on at the registry.</p>
<p>My image's name is this <code>gcr.io/kubehub-172120/simple</code>. Here's how I see the tags I have uploaded to <code>gcr.io</code>:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>gcloud<span class="w"> </span>beta<span class="w"> </span>container<span class="w"> </span>images<span class="w"> </span>list-tags<span class="w"> </span>gcr.io/kubehub-172120/simple
DIGEST<span class="w"> </span>TAGS<span class="w"> </span>TIMESTAMP
9b424f849df2<span class="w"> </span>88d38d9-debug,e8bc006<span class="w"> </span><span class="m">2017</span>-07-30T23:03:14
51cd80db604d<span class="w"> </span>12430ce<span class="w"> </span><span class="m">2017</span>-07-30T23:06:42
</code></pre></div>
<p>If you want to inspect the contents of the image, you can just run a shell, like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>gcloud<span class="w"> </span>docker<span class="w"> </span>--<span class="w"> </span>run<span class="w"> </span>-i<span class="w"> </span>-t<span class="w"> </span>gcr.io/kubehub-172120/simple:12430ce<span class="w"> </span>/bin/bash
root@27cfb042d947:/app#<span class="w"> </span>ls
Dockerfile<span class="w"> </span>app.py<span class="w"> </span>requirements.txt
root@27cfb042d947:/app#<span class="w"> </span>
</code></pre></div>
<p>I've used this workflow when working with a Kubernetes deployment and it worked well for me. Hope it's useful for someone else too. Enjoy.</p>Visualizing algorithms with Jupyter notebooks2017-04-24T21:10:00-07:002017-04-24T21:10:00-07:00Petko Minkovtag:pminkov.github.io,2017-04-24:/blog/visualizing-algorithms-with-jupyter-notebooks.html<p>Last couple of days I was looking into some old implementation of k-means clustering that I wrote. At that time I didn't know how to use matplotlib and to display the clusters that were generated and I was printing commands that I was plugging into R. Since then, I've learned …</p><p>Last couple of days I was looking into some old implementation of k-means clustering that I wrote. At that time I didn't know how to use matplotlib and to display the clusters that were generated and I was printing commands that I was plugging into R. Since then, I've learned how to use Jupyter, so I sat down and imported the code into a Jupyter notebook that's visualizing the algorithm.</p>
<p>Here's the <a href="http://nbviewer.jupyter.org/github/pminkov/notebooks/blob/master/k-means%20clustering.ipynb">Jupyter notebook</a>. Overall, I quite enjoy how this format mixes the results of code execution with Markdown segments.</p>Removing a node from a Kubernetes cluster on GKE (Google Container Engine)2017-03-30T16:17:00-07:002017-03-30T16:17:00-07:00Petko Minkovtag:pminkov.github.io,2017-03-30:/blog/removing-a-node-from-a-kubernetes-cluster-on-gke-google-container-engine.html<p>In this post, I'll describe how to remove a particular node from a Kubernetes cluster on GKE. Why would you want to do that? In my case, I'm running jupyterhub and I need to do that as part of implementing cluster scaling. That's probably a rare need, but it helped …</p><p>In this post, I'll describe how to remove a particular node from a Kubernetes cluster on GKE. Why would you want to do that? In my case, I'm running jupyterhub and I need to do that as part of implementing cluster scaling. That's probably a rare need, but it helped me understand more about the GCE structures behind a Kubernetes cluster.</p>
<p>So let's start. The first thing you need to do is:</p>
<h3 id="drain-your-node">Drain your node</h3>
<p>Let's look at my nodes:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kl<span class="w"> </span>get<span class="w"> </span>nodes
NAME<span class="w"> </span>STATUS<span class="w"> </span>AGE
gke-jcluster-default-pool-9cc4e660-rx9p<span class="w"> </span>Ready<span class="w"> </span>1d
gke-jcluster-default-pool-9cc4e660-xr4z<span class="w"> </span>Ready<span class="w"> </span>2d
</code></pre></div>
<p>I want to remove rx9p. I'll first <a href="https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/">drain</a> it:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kl<span class="w"> </span>drain<span class="w"> </span>gke-jcluster-default-pool-9cc4e660-rx9p<span class="w"> </span>--force
node<span class="w"> </span><span class="s2">"gke-jcluster-default-pool-9cc4e660-rx9p"</span><span class="w"> </span>cordoned
error:<span class="w"> </span>pods<span class="w"> </span>with<span class="w"> </span><span class="nb">local</span><span class="w"> </span>storage<span class="w"> </span><span class="o">(</span>use<span class="w"> </span>--delete-local-data<span class="w"> </span>to<span class="w"> </span>override<span class="o">)</span>:<span class="w"> </span>jupyter-petko-1
</code></pre></div>
<p>Great, the node is now drained. Next is:</p>
<h3 id="removing-the-gce-vm">Removing the GCE VM</h3>
<p>Your Kubernetes cluster runs in an <a href="https://cloud.google.com/compute/docs/instance-groups/">instance group</a>. We'll need to know what this group is. Here's how to do it from the command line.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">GROUP_ID</span><span class="o">=</span><span class="k">$(</span>gcloud<span class="w"> </span>container<span class="w"> </span>clusters<span class="w"> </span>describe<span class="w"> </span>jcluster<span class="w"> </span>--format<span class="w"> </span>json<span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>--raw-output<span class="w"> </span><span class="s1">'.instanceGroupUrls[0]'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>rev<span class="w"> </span><span class="p">|</span><span class="w"> </span>cut<span class="w"> </span>-d<span class="s1">'/'</span><span class="w"> </span>-f<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>rev<span class="k">)</span>
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$GROUP_ID</span>
gke-jcluster-default-pool-9cc4e660-grp
</code></pre></div>
<p>Let's check my instances:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>gcloud<span class="w"> </span>compute<span class="w"> </span>instances<span class="w"> </span>list
NAME<span class="w"> </span>ZONE<span class="w"> </span>MACHINE_TYPE<span class="w"> </span>PREEMPTIBLE<span class="w"> </span>INTERNAL_IP<span class="w"> </span>EXTERNAL_IP<span class="w"> </span>STATUS
gke-jcluster-default-pool-9cc4e660-rx9p<span class="w"> </span>us-central1-b<span class="w"> </span>n1-standard-1<span class="w"> </span><span class="m">10</span>.128.0.2<span class="w"> </span><span class="m">104</span>.198.174.222<span class="w"> </span>RUNNING
gke-jcluster-default-pool-9cc4e660-xr4z<span class="w"> </span>us-central1-b<span class="w"> </span>n1-standard-1<span class="w"> </span><span class="m">10</span>.128.0.4<span class="w"> </span><span class="m">104</span>.197.237.135<span class="w"> </span>RUNNING
</code></pre></div>
<p>If I just run <code>gcloud compute instances delete</code> that won't work! That's because I have an instance group of size 2 and if I delete one of the machines, GCE will start a new one. I have to use the <code>gcloud compute instance-groups managed delete-instances</code> command, followed by <code>gcloud compute instance-groups managed wait-until-stable</code> if I want to wait until the job is done.</p>
<p>Let's see how that looks like:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>gcloud<span class="w"> </span>compute<span class="w"> </span>instance-groups<span class="w"> </span>managed<span class="w"> </span>delete-instances<span class="w"> </span><span class="nv">$GROUP_ID</span><span class="w"> </span>--instances<span class="o">=</span>gke-jcluster-default-pool-9cc4e660-rx9p
Updated<span class="w"> </span><span class="o">[</span>https://www.googleapis.com/compute/v1/projects/myhub-161019/zones/us-central1-b/instanceGroupManagers/gke-jcluster-default-pool-9cc4e660-grp<span class="o">]</span>.
---
baseInstanceName:<span class="w"> </span>gke-jcluster-default-pool-9cc4e660
creationTimestamp:<span class="w"> </span><span class="s1">'2017-03-25T02:52:22.040-07:00'</span>
currentActions:
<span class="w"> </span>abandoning:<span class="w"> </span><span class="m">0</span>
<span class="w"> </span>creating:<span class="w"> </span><span class="m">0</span>
<span class="w"> </span>creatingWithoutRetries:<span class="w"> </span><span class="m">0</span>
<span class="w"> </span>deleting:<span class="w"> </span><span class="m">1</span>
<span class="w"> </span>none:<span class="w"> </span><span class="m">1</span>
<span class="w"> </span>recreating:<span class="w"> </span><span class="m">0</span>
<span class="w"> </span>refreshing:<span class="w"> </span><span class="m">0</span>
<span class="w"> </span>restarting:<span class="w"> </span><span class="m">0</span>
fingerprint:<span class="w"> </span><span class="nv">kUg7ggCEudY</span><span class="o">=</span>
id:<span class="w"> </span><span class="s1">'6475008099735012154'</span>
instanceGroup:<span class="w"> </span>gke-jcluster-default-pool-9cc4e660-grp
instanceTemplate:<span class="w"> </span>gke-jcluster-default-pool-9cc4e660
kind:<span class="w"> </span>compute#instanceGroupManager
name:<span class="w"> </span>gke-jcluster-default-pool-9cc4e660-grp
selfLink:<span class="w"> </span>https://www.googleapis.com/compute/v1/projects/myhub-161019/zones/us-central1-b/instanceGroupManagers/gke-jcluster-default-pool-9cc4e660-grp
targetSize:<span class="w"> </span><span class="m">1</span>
zone:<span class="w"> </span>us-central1-b
$<span class="w"> </span>gcloud<span class="w"> </span>compute<span class="w"> </span>instance-groups<span class="w"> </span>managed<span class="w"> </span>wait-until-stable<span class="w"> </span><span class="nv">$GROUP_ID</span>
Waiting<span class="w"> </span><span class="k">for</span><span class="w"> </span>group<span class="w"> </span>to<span class="w"> </span>become<span class="w"> </span>stable,<span class="w"> </span>current<span class="w"> </span>operations:<span class="w"> </span>deleting:<span class="w"> </span><span class="m">1</span>
...
Waiting<span class="w"> </span><span class="k">for</span><span class="w"> </span>group<span class="w"> </span>to<span class="w"> </span>become<span class="w"> </span>stable,<span class="w"> </span>current<span class="w"> </span>operations:<span class="w"> </span>deleting:<span class="w"> </span><span class="m">1</span>
Group<span class="w"> </span>is<span class="w"> </span>stable
</code></pre></div>
<p>And indeed, we only have one node left now:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kl<span class="w"> </span>get<span class="w"> </span>nodes
gcloud<span class="w"> </span>compute<span class="w"> </span>instanceNAME<span class="w"> </span>STATUS<span class="w"> </span>AGE
gke-jcluster-default-pool-9cc4e660-xr4z<span class="w"> </span>Ready<span class="w"> </span>2d
$<span class="w"> </span>gcloud<span class="w"> </span>compute<span class="w"> </span>instances<span class="w"> </span>list
NAME<span class="w"> </span>ZONE<span class="w"> </span>MACHINE_TYPE<span class="w"> </span>PREEMPTIBLE<span class="w"> </span>INTERNAL_IP<span class="w"> </span>EXTERNAL_IP<span class="w"> </span>STATUS
gke-jcluster-default-pool-9cc4e660-xr4z<span class="w"> </span>us-central1-b<span class="w"> </span>n1-standard-1<span class="w"> </span><span class="m">10</span>.128.0.4<span class="w"> </span><span class="m">104</span>.197.237.135<span class="w"> </span>RUNNING
</code></pre></div>
<p>So that's it. Regardless of whether you need to delete individual nodes, it's interesting to take a look at how you can do that.</p>Starting with Kubernetes on Google Container Engine2017-03-07T12:42:00-08:002017-03-07T12:42:00-08:00Petko Minkovtag:pminkov.github.io,2017-03-07:/blog/starting-with-kubernetes-on-google-container-engine.html<p>This is a tutorial of how to run a simple Kubernetes app on <a href="https://cloud.google.com/container-engine/">GKE</a> (Google Container Engine).</p>
<p><a href="https://kubernetes.io/">Kubernetes</a> is a container orchestration software that started at Google. Right now it's an open source project maintained by <a href="https://www.cncf.io/">Cloud Native Computing Foundation</a>. </p>
<p>So what's Kubernetes all about? Roughly, it's about managing a …</p><p>This is a tutorial of how to run a simple Kubernetes app on <a href="https://cloud.google.com/container-engine/">GKE</a> (Google Container Engine).</p>
<p><a href="https://kubernetes.io/">Kubernetes</a> is a container orchestration software that started at Google. Right now it's an open source project maintained by <a href="https://www.cncf.io/">Cloud Native Computing Foundation</a>. </p>
<p>So what's Kubernetes all about? Roughly, it's about managing a cluster (a set of machines) on which we have containers running. </p>
<h3 id="why-containers">Why containers?</h3>
<p>Well, that's a question that probably has a lot of answers, but containers are very lightweight virtual machines more or less. The technology under them is different than virtual machines though. We can orchestrate a set of VMs, but containers are much easier to work with. For example I have one VM running on my MacBook - it eats a lot of memory, it's managed by VirtualBox, it has a lot of stuff installed on it. I can't imagine having to run a bunch of these. Containers though, they're much easier to setup (you just write a text file that describes them - the Dockerfile) and much more lightweight. Their memory footprint can be pretty low. Containers can run on different OSs, some of which are very lightweight. Ok, enough about containers - back to Kubernetes.</p>
<h3 id="whats-in-a-cluster">What's in a cluster?</h3>
<p>I mentioned clusters. But what's a cluster. A cluster has a set of nodes (think computers) on which we have pods, where each pod is a unit that contains several containers. That's the basic structure.</p>
<h3 id="running-a-simple-kubernetes-app-on-gke">Running a simple Kubernetes app on GKE.</h3>
<p>Alright, let's roll our sleeves. Google has its own <a href="https://cloud.google.com/container-engine/docs/quickstart">Quickstart</a> tutorial, but what I don't like about it is that it doesn't describe how to create your own container and it doesn't talk about the Kubernetes dashboard. But a lot of the steps here you can see in Google's tutorial as well.</p>
<p>So let's start.</p>
<h3 id="create-a-project">Create a project.</h3>
<ol>
<li>Go to the <a href="https://console.cloud.google.com">Cloud Console</a>.</li>
<li>Create a project. Mine is called "mykube". Every project has an id that you're mostly working with. Mine is <code>mykube-160819</code>.</li>
</ol>
<h3 id="install-necessary-tools-and-initialize-them">Install necessary tools and initialize them.</h3>
<ol>
<li>Install the <a href="https://cloud.google.com/sdk/downloads">Google Cloud SDK</a>. There's a web interface for working with the SDK, called Google Cloud Shell, but I like having the tools installed locally.</li>
<li>Initialize gcloud by running <code>gcloud init</code>. You'll be asked for the name of your project.</li>
<li>Set a <a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones#available">Compute Engine zone</a>, like this:</li>
</ol>
<div class="highlight"><pre><span></span><code>gcloud<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>compute/zone<span class="w"> </span>us-central1-b
</code></pre></div>
<p>That's it. You can view your configuration:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>gcloud<span class="w"> </span>config<span class="w"> </span>list
Your<span class="w"> </span>active<span class="w"> </span>configuration<span class="w"> </span>is:<span class="w"> </span><span class="o">[</span>mykube<span class="o">]</span>
<span class="o">[</span>compute<span class="o">]</span>
<span class="nv">zone</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>us-central1-b
<span class="o">[</span>core<span class="o">]</span>
<span class="nv">account</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>pminkov@gmail.com
<span class="nv">disable_usage_reporting</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>False
<span class="nv">project</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>mykube-160819
</code></pre></div>
<p>Let's authenticate gcloud too:</p>
<div class="highlight"><pre><span></span><code>gcloud<span class="w"> </span>auth<span class="w"> </span>application-default<span class="w"> </span>login
</code></pre></div>
<h3 id="run-a-container-image">Run a container image.</h3>
<p>Our container is going to be a node.js application that we'll build ourselves. I wanted to experiment with an app that takes a bit more memory, so here's how my code looks like:</p>
<div class="highlight"><pre><span></span><code><span class="nx">$</span><span class="w"> </span><span class="nx">cat</span><span class="w"> </span><span class="p">.</span><span class="o">/</span><span class="nx">server</span><span class="p">.</span><span class="nx">js</span><span class="w"> </span>
<span class="kd">function</span><span class="w"> </span><span class="nx">randomInt</span><span class="p">(</span><span class="nx">n</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Math</span><span class="p">.</span><span class="nx">floor</span><span class="p">(</span><span class="nb">Math</span><span class="p">.</span><span class="nx">random</span><span class="p">()</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="nx">n</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">http</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">require</span><span class="p">(</span><span class="s1">'http'</span><span class="p">);</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">N</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">20000000</span><span class="p">;</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">nums</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nb">Array</span><span class="p">(</span><span class="nx">N</span><span class="p">);</span>
<span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="kd">var</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">0</span><span class="p">;</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="nx">N</span><span class="p">;</span><span class="w"> </span><span class="nx">i</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">nums</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">randomInt</span><span class="p">(</span><span class="nx">N</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mf">10</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">handleRequest</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kd">function</span><span class="p">(</span><span class="nx">request</span><span class="p">,</span><span class="w"> </span><span class="nx">response</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Received request for URL: '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">request</span><span class="p">.</span><span class="nx">url</span><span class="p">);</span>
<span class="w"> </span><span class="nx">response</span><span class="p">.</span><span class="nx">writeHead</span><span class="p">(</span><span class="mf">200</span><span class="p">);</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">index</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">randomInt</span><span class="p">(</span><span class="nx">N</span><span class="p">);</span>
<span class="w"> </span><span class="nx">response</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="s1">'Returning element at index '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">index</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">': '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">nums</span><span class="p">[</span><span class="nx">index</span><span class="p">]</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">'\n'</span><span class="p">);</span>
<span class="w"> </span><span class="nx">response</span><span class="p">.</span><span class="nx">end</span><span class="p">(</span><span class="s1">'Hello World!'</span><span class="p">);</span>
<span class="p">};</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">www</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">createServer</span><span class="p">(</span><span class="nx">handleRequest</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Listening on port 8080'</span><span class="p">);</span>
<span class="nx">www</span><span class="p">.</span><span class="nx">listen</span><span class="p">(</span><span class="mf">8080</span><span class="p">);</span>
</code></pre></div>
<p>We also need a Dockerfile:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>./Dockerfile<span class="w"> </span>
FROM<span class="w"> </span>node:4
EXPOSE<span class="w"> </span><span class="m">8080</span>
COPY<span class="w"> </span>server.js<span class="w"> </span>.
CMD<span class="w"> </span>node<span class="w"> </span>server.js
</code></pre></div>
<p>In order to run a container on GKE, we need to upload it to <a href="https://cloud.google.com/container-registry/">Google Container Registry</a>. Let's do it:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PROJECT_ID</span><span class="o">=</span>mykube-160819
$<span class="w"> </span>docker<span class="w"> </span>build<span class="w"> </span>-t<span class="w"> </span>gcr.io/<span class="nv">$PROJECT_ID</span>/myserver<span class="w"> </span>.
Sending<span class="w"> </span>build<span class="w"> </span>context<span class="w"> </span>to<span class="w"> </span>Docker<span class="w"> </span>daemon<span class="w"> </span><span class="m">3</span>.584<span class="w"> </span>kB
Step<span class="w"> </span><span class="m">1</span>/4<span class="w"> </span>:<span class="w"> </span>FROM<span class="w"> </span>node:4
<span class="w"> </span>---><span class="w"> </span>d7efee1f035d
Step<span class="w"> </span><span class="m">2</span>/4<span class="w"> </span>:<span class="w"> </span>EXPOSE<span class="w"> </span><span class="m">8080</span>
<span class="w"> </span>---><span class="w"> </span>Using<span class="w"> </span>cache
<span class="w"> </span>---><span class="w"> </span>147e7888542d
Step<span class="w"> </span><span class="m">3</span>/4<span class="w"> </span>:<span class="w"> </span>COPY<span class="w"> </span>server.js<span class="w"> </span>.
<span class="w"> </span>---><span class="w"> </span>Using<span class="w"> </span>cache
<span class="w"> </span>---><span class="w"> </span>b610e7975d20
Step<span class="w"> </span><span class="m">4</span>/4<span class="w"> </span>:<span class="w"> </span>CMD<span class="w"> </span>node<span class="w"> </span>server.js
<span class="w"> </span>---><span class="w"> </span>Using<span class="w"> </span>cache
<span class="w"> </span>---><span class="w"> </span>4e15133cdab2
Successfully<span class="w"> </span>built<span class="w"> </span>4e15133cdab2
$<span class="w"> </span>gcloud<span class="w"> </span>docker<span class="w"> </span>--<span class="w"> </span>push<span class="w"> </span>gcr.io/<span class="nv">$PROJECT_ID</span>/myserver
The<span class="w"> </span>push<span class="w"> </span>refers<span class="w"> </span>to<span class="w"> </span>a<span class="w"> </span>repository<span class="w"> </span><span class="o">[</span>gcr.io/mykube-160819/myserver<span class="o">]</span>
<span class="c1"># Note that I already have a project called "hikube" which has the same docker image.</span>
<span class="c1"># The "mykube" project is something I created for this tutorial.</span>
1bcf3881e79d:<span class="w"> </span>Mounted<span class="w"> </span>from<span class="w"> </span>hikube-160719/myserver<span class="w"> </span>
65e403c25ee9:<span class="w"> </span>Mounted<span class="w"> </span>from<span class="w"> </span>hikube-160719/myserver<span class="w"> </span>
4732c3666dd7:<span class="w"> </span>Mounted<span class="w"> </span>from<span class="w"> </span>hikube-160719/myserver<span class="w"> </span>
a1fbf6fa923f:<span class="w"> </span>Mounted<span class="w"> </span>from<span class="w"> </span>hikube-160719/myserver<span class="w"> </span>
1b8ef9ac5116:<span class="w"> </span>Mounted<span class="w"> </span>from<span class="w"> </span>hikube-160719/myserver<span class="w"> </span>
41ef8cc0bccb:<span class="w"> </span>Mounted<span class="w"> </span>from<span class="w"> </span>hikube-160719/myserver<span class="w"> </span>
100396c46221:<span class="w"> </span>Mounted<span class="w"> </span>from<span class="w"> </span>hikube-160719/myserver<span class="w"> </span>
7b4b54c74241:<span class="w"> </span>Mounted<span class="w"> </span>from<span class="w"> </span>hikube-160719/myserver<span class="w"> </span>
d17d48b2382a:<span class="w"> </span>Mounted<span class="w"> </span>from<span class="w"> </span>hikube-160719/myserver<span class="w"> </span>
latest:<span class="w"> </span>digest:<span class="w"> </span>sha256:4ad68f056e870938823f6c9555355c149cf7c42a213d7243d915f1a4bcfb9cb1<span class="w"> </span>size:<span class="w"> </span><span class="m">2213</span>
</code></pre></div>
<p>If you go to the Google Cloud Console and open the Google Container Registry, you'll see the container uploaded:</p>
<p><center><img alt="Google Container Registry" src="https://pminkov.github.io/blog/images/k8s-intro/k1-registry.png"></center></p>
<p>Now let's create a cluster that we'll be deploying our server to:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>gcloud<span class="w"> </span>container<span class="w"> </span>clusters<span class="w"> </span>create<span class="w"> </span>example-cluster
Creating<span class="w"> </span>cluster<span class="w"> </span>example-cluster...done.<span class="w"> </span>
Created<span class="w"> </span><span class="o">[</span>https://container.googleapis.com/v1/projects/mykube-160819/zones/us-central1-b/clusters/example-cluster<span class="o">]</span>.
kubeconfig<span class="w"> </span>entry<span class="w"> </span>generated<span class="w"> </span><span class="k">for</span><span class="w"> </span>example-cluster.
NAME<span class="w"> </span>ZONE<span class="w"> </span>MASTER_VERSION<span class="w"> </span>MASTER_IP<span class="w"> </span>MACHINE_TYPE<span class="w"> </span>NODE_VERSION<span class="w"> </span>NUM_NODES<span class="w"> </span>STATUS
example-cluster<span class="w"> </span>us-central1-b<span class="w"> </span><span class="m">1</span>.5.3<span class="w"> </span><span class="m">104</span>.198.190.52<span class="w"> </span>n1-standard-1<span class="w"> </span><span class="m">1</span>.5.3<span class="w"> </span><span class="m">3</span><span class="w"> </span>RUNNING
</code></pre></div>
<p>Congratulations! We have a cluster running.</p>
<p>A little segway. kubectl has a config that determines which cluster you're working with. You can switch between different clusters. Try it:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kubectl<span class="w"> </span>config<span class="w"> </span>current-context
gke_mykube-160819_us-central1-b_example-cluster
</code></pre></div>
<p>We're good here - our context is for the "mykube" cluster. The cluster is empty, we can verify it like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kubectl<span class="w"> </span>get<span class="w"> </span>pods
No<span class="w"> </span>resources<span class="w"> </span>found.
</code></pre></div>
<p>Now let's start out server finally:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kubectl<span class="w"> </span>run<span class="w"> </span>myserver<span class="w"> </span>--image<span class="o">=</span>gcr.io/<span class="nv">$PROJECT_ID</span>/myserver<span class="w"> </span>--port<span class="o">=</span><span class="m">8080</span>
deployment<span class="w"> </span><span class="s2">"myserver"</span><span class="w"> </span>created
</code></pre></div>
<p>We have created a deployment that contains a pod. Let's see what pods we have now:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kubectl<span class="w"> </span>get<span class="w"> </span>pods
NAME<span class="w"> </span>READY<span class="w"> </span>STATUS<span class="w"> </span>RESTARTS<span class="w"> </span>AGE
myserver-3430466764-04b36<span class="w"> </span><span class="m">0</span>/1<span class="w"> </span>ContainerCreating<span class="w"> </span><span class="m">0</span><span class="w"> </span>17s
</code></pre></div>
<p>Nice, our container is getting spinned. We wait for a bit and we see:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kubectl<span class="w"> </span>get<span class="w"> </span>pods
NAME<span class="w"> </span>READY<span class="w"> </span>STATUS<span class="w"> </span>RESTARTS<span class="w"> </span>AGE
myserver-3430466764-04b36<span class="w"> </span><span class="m">1</span>/1<span class="w"> </span>Running<span class="w"> </span><span class="m">0</span><span class="w"> </span>58s
</code></pre></div>
<p>We can now expose the container:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kubectl<span class="w"> </span>expose<span class="w"> </span>deployment<span class="w"> </span>myserver<span class="w"> </span>--type<span class="o">=</span><span class="s2">"LoadBalancer"</span>
service<span class="w"> </span><span class="s2">"myserver"</span><span class="w"> </span>exposed
</code></pre></div>
<p>This will also take some time:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kubectl<span class="w"> </span>get<span class="w"> </span>service<span class="w"> </span>myserver
NAME<span class="w"> </span>CLUSTER-IP<span class="w"> </span>EXTERNAL-IP<span class="w"> </span>PORT<span class="o">(</span>S<span class="o">)</span><span class="w"> </span>AGE
myserver<span class="w"> </span><span class="m">10</span>.3.247.6<span class="w"> </span><pending><span class="w"> </span><span class="m">8080</span>:31574/TCP<span class="w"> </span>34s
</code></pre></div>
<p>Aaand, it's done:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kubectl<span class="w"> </span>get<span class="w"> </span>service<span class="w"> </span>myserver
NAME<span class="w"> </span>CLUSTER-IP<span class="w"> </span>EXTERNAL-IP<span class="w"> </span>PORT<span class="o">(</span>S<span class="o">)</span><span class="w"> </span>AGE
myserver<span class="w"> </span><span class="m">10</span>.3.247.6<span class="w"> </span><span class="m">104</span>.155.177.47<span class="w"> </span><span class="m">8080</span>:31574/TCP<span class="w"> </span>1m
</code></pre></div>
<p>If we go to <code>http://104.155.177.47:8080/</code>, we'll see:</p>
<div class="highlight"><pre><span></span><code>Returning<span class="w"> </span>element<span class="w"> </span>at<span class="w"> </span>index<span class="w"> </span><span class="m">6110645</span>:<span class="w"> </span><span class="m">116527640</span>
Hello<span class="w"> </span>World!
</code></pre></div>
<p>Now, the tedious part is over. Let's have some fun. We can monitor our cluster through the Kubernetes dashboard. For a reason unclear to me, this dashboard is not available on the Google website. You have to run a proxy to do it. Like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kubectl<span class="w"> </span>proxy
Starting<span class="w"> </span>to<span class="w"> </span>serve<span class="w"> </span>on<span class="w"> </span><span class="m">127</span>.0.0.1:8001
</code></pre></div>
<p>We can see the dashboard at <code>localhost:8001/ui</code>. It looks like this:</p>
<p><center><img alt="Kubernetes Dashboard" src="https://pminkov.github.io/blog/images/k8s-intro/k2-dashboard.png"></center></p>
<p>This dashboard is a lot of fun. You can dig into everything available in it. You can probably see everything it shows through kubectl as well, but it's easier to do it by using an UI.</p>
<p>Here's something else that's fun. Your cluster doesn't run on thin air. It runs on Google Compute Engine instances (I believe this is equivalent to AWS' EC2). In your cloud console, you can navigate to your instances and you can even SSH to an instance from the web UI (I wow-ed the first time I did this, much easier than setting up ssh access on AWS).</p>
<p>Our cluster has three nodes.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kubectl<span class="w"> </span>get<span class="w"> </span>nodes
NAME<span class="w"> </span>STATUS<span class="w"> </span>AGE
gke-example-cluster-default-pool-2567fc65-1h40<span class="w"> </span>Ready<span class="w"> </span>21m
gke-example-cluster-default-pool-2567fc65-g7lc<span class="w"> </span>Ready<span class="w"> </span>21m
gke-example-cluster-default-pool-2567fc65-n0cp<span class="w"> </span>Ready<span class="w"> </span>21m
</code></pre></div>
<p>These are three GCE instances. Where is our pod running at?</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kubectl<span class="w"> </span>get<span class="w"> </span>pods<span class="w"> </span>-o<span class="w"> </span>wide
NAME<span class="w"> </span>READY<span class="w"> </span>STATUS<span class="w"> </span>RESTARTS<span class="w"> </span>AGE<span class="w"> </span>IP<span class="w"> </span>NODE
myserver-3430466764-04b36<span class="w"> </span><span class="m">1</span>/1<span class="w"> </span>Running<span class="w"> </span><span class="m">0</span><span class="w"> </span>18m<span class="w"> </span><span class="m">10</span>.0.1.5<span class="w"> </span>gke-example-cluster-default-pool-2567fc65-g7lc
</code></pre></div>
<p>It's running on <code>gke-example-cluster-default-pool-2567fc65-g7lc</code>. Now I can navigate to the web page for this instance and ssh to it. Here's a screenshot of how that looks like:</p>
<p><center><img alt="Compute Engine SSH" src="https://pminkov.github.io/blog/images/k8s-intro/k3-compute-engine.png"></center></p>
<p>I ran <code>ps aux --sort '%mem'</code> to see which process takes most memory. Since my server uses a lot of memory, it's at the top. It's using 179MB of resident memory.</p>
<p>It's pretty nice that you're able to nagivate from a high level system like Kubernetes all the way down to ssh-ing to a machine that runs your containers. When you're ssh-ed you can execute <code>docker ps</code> to see what containers are running, run <code>top</code> to see what's going on on the machine and do all of the other systems debugging tasks that you can think of.</p>
<p>And finally, let's delete our cluster:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>gcloud<span class="w"> </span>container<span class="w"> </span>clusters<span class="w"> </span>delete<span class="w"> </span>example-cluster
The<span class="w"> </span>following<span class="w"> </span>clusters<span class="w"> </span>will<span class="w"> </span>be<span class="w"> </span>deleted.
<span class="w"> </span>-<span class="w"> </span><span class="o">[</span>example-cluster<span class="o">]</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="o">[</span>us-central1-b<span class="o">]</span>
Do<span class="w"> </span>you<span class="w"> </span>want<span class="w"> </span>to<span class="w"> </span><span class="k">continue</span><span class="w"> </span><span class="o">(</span>Y/n<span class="o">)</span>?<span class="w"> </span>
Deleting<span class="w"> </span>cluster<span class="w"> </span>example-cluster...done.<span class="w"> </span>
Deleted<span class="w"> </span><span class="o">[</span>https://container.googleapis.com/v1/projects/mykube-160819/zones/us-central1-b/clusters/example-cluster<span class="o">]</span>.
</code></pre></div>
<p>That's all for today - enjoy.</p>Examining a process in Linux.2017-03-01T12:26:00-08:002017-03-01T12:26:00-08:00Petko Minkovtag:pminkov.github.io,2017-03-01:/blog/examining-a-process-in-linux.html<p>I've been thinking about writing a blog post about Linux tools and commands related to processes. Let's take a look at some of them.</p>
<p>The process that we'll be looking at is a <a href="https://github.com/pminkov/webserver">webserver</a> that I wrote some time ago to practice my C and write some code that does …</p><p>I've been thinking about writing a blog post about Linux tools and commands related to processes. Let's take a look at some of them.</p>
<p>The process that we'll be looking at is a <a href="https://github.com/pminkov/webserver">webserver</a> that I wrote some time ago to practice my C and write some code that does network related work. This webserver runs a <a href="https://github.com/pminkov/threadpool">threadpool</a> where N threads are waiting for server requests that they're going to execute.</p>
<p>So let's start the server:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>./server
Running<span class="w"> </span>on<span class="w"> </span>port:<span class="w"> </span><span class="m">8000</span>
</code></pre></div>
<p>Great. So which process is this server running as? We can use the <code>pidof</code> command to find that out. Its output looks like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pidof<span class="w"> </span>server
<span class="m">8876</span>
</code></pre></div>
<p>If we had other processes which were running an executable with that name, we'd see more process ids, but since we only have one, we see one process id.</p>
<p>What next? Let's see how the process is layed out in memory. To do this, we can use the <code>pmap</code> command. Its output looks like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>pmap<span class="w"> </span>-p<span class="w"> </span><span class="m">8876</span>
<span class="m">8876</span>:<span class="w"> </span>./server
<span class="m">0000000000400000</span><span class="w"> </span>16K<span class="w"> </span>r-x--<span class="w"> </span>/home/petko/work/github/webserver/server
<span class="m">0000000000603000</span><span class="w"> </span>4K<span class="w"> </span>r----<span class="w"> </span>/home/petko/work/github/webserver/server
<span class="m">0000000000604000</span><span class="w"> </span>4K<span class="w"> </span>rw---<span class="w"> </span>/home/petko/work/github/webserver/server
000000000110f000<span class="w"> </span>132K<span class="w"> </span>rw---<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5ca731000<span class="w"> </span>4K<span class="w"> </span>-----<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5ca732000<span class="w"> </span>8192K<span class="w"> </span>rw---<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5caf32000<span class="w"> </span>4K<span class="w"> </span>-----<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5caf33000<span class="w"> </span>8192K<span class="w"> </span>rw---<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5cb733000<span class="w"> </span>4K<span class="w"> </span>-----<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5cb734000<span class="w"> </span>8192K<span class="w"> </span>rw---<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5cbf34000<span class="w"> </span>4K<span class="w"> </span>-----<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5cbf35000<span class="w"> </span>8192K<span class="w"> </span>rw---<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5cc735000<span class="w"> </span>1792K<span class="w"> </span>r-x--<span class="w"> </span>/lib/x86_64-linux-gnu/libc-2.23.so
00007fd5cc8f5000<span class="w"> </span>2044K<span class="w"> </span>-----<span class="w"> </span>/lib/x86_64-linux-gnu/libc-2.23.so
00007fd5ccaf4000<span class="w"> </span>16K<span class="w"> </span>r----<span class="w"> </span>/lib/x86_64-linux-gnu/libc-2.23.so
00007fd5ccaf8000<span class="w"> </span>8K<span class="w"> </span>rw---<span class="w"> </span>/lib/x86_64-linux-gnu/libc-2.23.so
00007fd5ccafa000<span class="w"> </span>16K<span class="w"> </span>rw---<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5ccafe000<span class="w"> </span>96K<span class="w"> </span>r-x--<span class="w"> </span>/lib/x86_64-linux-gnu/libpthread-2.23.so
00007fd5ccb16000<span class="w"> </span>2044K<span class="w"> </span>-----<span class="w"> </span>/lib/x86_64-linux-gnu/libpthread-2.23.so
00007fd5ccd15000<span class="w"> </span>4K<span class="w"> </span>r----<span class="w"> </span>/lib/x86_64-linux-gnu/libpthread-2.23.so
00007fd5ccd16000<span class="w"> </span>4K<span class="w"> </span>rw---<span class="w"> </span>/lib/x86_64-linux-gnu/libpthread-2.23.so
00007fd5ccd17000<span class="w"> </span>16K<span class="w"> </span>rw---<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5ccd1b000<span class="w"> </span>152K<span class="w"> </span>r-x--<span class="w"> </span>/lib/x86_64-linux-gnu/ld-2.23.so
00007fd5ccf22000<span class="w"> </span>12K<span class="w"> </span>rw---<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5ccf3e000<span class="w"> </span>8K<span class="w"> </span>rw---<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007fd5ccf40000<span class="w"> </span>4K<span class="w"> </span>r----<span class="w"> </span>/lib/x86_64-linux-gnu/ld-2.23.so
00007fd5ccf41000<span class="w"> </span>4K<span class="w"> </span>rw---<span class="w"> </span>/lib/x86_64-linux-gnu/ld-2.23.so
00007fd5ccf42000<span class="w"> </span>4K<span class="w"> </span>rw---<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007ffca0861000<span class="w"> </span>132K<span class="w"> </span>rw---<span class="w"> </span><span class="o">[</span><span class="w"> </span>stack<span class="w"> </span><span class="o">]</span>
00007ffca09eb000<span class="w"> </span>8K<span class="w"> </span>r----<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
00007ffca09ed000<span class="w"> </span>8K<span class="w"> </span>r-x--<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
ffffffffff600000<span class="w"> </span>4K<span class="w"> </span>r-x--<span class="w"> </span><span class="o">[</span><span class="w"> </span>anon<span class="w"> </span><span class="o">]</span>
total<span class="w"> </span>39316K
</code></pre></div>
<p>What you see here are virtual memory addresses. For example, let's take a look at this line:</p>
<p><code>00007fd5cc735000 1792K r-x-- /lib/x86_64-linux-gnu/libc-2.23.so</code></p>
<p>This is the code for <code>libc</code>, which is the C standard library. This code is shared between processes that need it. We can see the <code>x</code> flag, which means that this is executable memory. The size if roughly the same as the size of this <code>so</code> file. This library is memory mapped into a region starting at address <code>00007fd5cc735000</code>, but in physical memory it's only stored in one place. To learn more about memory in Linux, here's a <a href="https://techtalk.intersec.com/2013/07/memory-part-1-memory-types/">great post</a> going into detail about it.</p>
<p>Another interesting command is <code>lsof</code>. <code>lsof</code> stands for "list of open files". Let's see its output:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>lsof<span class="w"> </span>-p<span class="w"> </span><span class="m">8876</span>
COMMAND<span class="w"> </span>PID<span class="w"> </span>USER<span class="w"> </span>FD<span class="w"> </span>TYPE<span class="w"> </span>DEVICE<span class="w"> </span>SIZE/OFF<span class="w"> </span>NODE<span class="w"> </span>NAME
server<span class="w"> </span><span class="m">8876</span><span class="w"> </span>petko<span class="w"> </span>cwd<span class="w"> </span>DIR<span class="w"> </span><span class="m">8</span>,1<span class="w"> </span><span class="m">4096</span><span class="w"> </span><span class="m">262299</span><span class="w"> </span>/home/petko/work/github/webserver
server<span class="w"> </span><span class="m">8876</span><span class="w"> </span>petko<span class="w"> </span>rtd<span class="w"> </span>DIR<span class="w"> </span><span class="m">8</span>,1<span class="w"> </span><span class="m">4096</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>/
server<span class="w"> </span><span class="m">8876</span><span class="w"> </span>petko<span class="w"> </span>txt<span class="w"> </span>REG<span class="w"> </span><span class="m">8</span>,1<span class="w"> </span><span class="m">25536</span><span class="w"> </span><span class="m">306491</span><span class="w"> </span>/home/petko/work/github/webserver/server
server<span class="w"> </span><span class="m">8876</span><span class="w"> </span>petko<span class="w"> </span>mem<span class="w"> </span>REG<span class="w"> </span><span class="m">8</span>,1<span class="w"> </span><span class="m">1864888</span><span class="w"> </span><span class="m">1184834</span><span class="w"> </span>/lib/x86_64-linux-gnu/libc-2.23.so
server<span class="w"> </span><span class="m">8876</span><span class="w"> </span>petko<span class="w"> </span>mem<span class="w"> </span>REG<span class="w"> </span><span class="m">8</span>,1<span class="w"> </span><span class="m">138744</span><span class="w"> </span><span class="m">1184980</span><span class="w"> </span>/lib/x86_64-linux-gnu/libpthread-2.23.so
server<span class="w"> </span><span class="m">8876</span><span class="w"> </span>petko<span class="w"> </span>mem<span class="w"> </span>REG<span class="w"> </span><span class="m">8</span>,1<span class="w"> </span><span class="m">162632</span><span class="w"> </span><span class="m">1184806</span><span class="w"> </span>/lib/x86_64-linux-gnu/ld-2.23.so
server<span class="w"> </span><span class="m">8876</span><span class="w"> </span>petko<span class="w"> </span>0u<span class="w"> </span>CHR<span class="w"> </span><span class="m">136</span>,9<span class="w"> </span>0t0<span class="w"> </span><span class="m">12</span><span class="w"> </span>/dev/pts/9
server<span class="w"> </span><span class="m">8876</span><span class="w"> </span>petko<span class="w"> </span>1u<span class="w"> </span>CHR<span class="w"> </span><span class="m">136</span>,9<span class="w"> </span>0t0<span class="w"> </span><span class="m">12</span><span class="w"> </span>/dev/pts/9
server<span class="w"> </span><span class="m">8876</span><span class="w"> </span>petko<span class="w"> </span>2u<span class="w"> </span>CHR<span class="w"> </span><span class="m">136</span>,9<span class="w"> </span>0t0<span class="w"> </span><span class="m">12</span><span class="w"> </span>/dev/pts/9
server<span class="w"> </span><span class="m">8876</span><span class="w"> </span>petko<span class="w"> </span>3u<span class="w"> </span>IPv4<span class="w"> </span><span class="m">81993</span><span class="w"> </span>0t0<span class="w"> </span>TCP<span class="w"> </span>*:8000<span class="w"> </span><span class="o">(</span>LISTEN<span class="o">)</span>
</code></pre></div>
<p>As you can see, we have file descriptors 0,1 and 2, which are stdin, stdout and stderr. They are linked to the terminal in which the process is running in. You can write to that terminal btw. Just type <code>echo "hello world" > /dev/pts/9</code> and you'll see that text in the terminal where your webserver is running. File descriptor number 3 is our socket which accepts connections.</p>
<p>Another interesting way to inspect processes is the ps command. Its basic output looks like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ps<span class="w"> </span>--pid<span class="w"> </span><span class="m">8876</span>
<span class="w"> </span>PID<span class="w"> </span>TTY<span class="w"> </span>TIME<span class="w"> </span>CMD
<span class="w"> </span><span class="m">8876</span><span class="w"> </span>pts/9<span class="w"> </span><span class="m">00</span>:00:00<span class="w"> </span>server
</code></pre></div>
<p>This is simple. We can also show the threads inside a process, like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ps<span class="w"> </span>m<span class="w"> </span>--pid<span class="w"> </span><span class="m">8876</span><span class="w"> </span>-o<span class="w"> </span>pid,tid,cmd
<span class="w"> </span>PID<span class="w"> </span>TID<span class="w"> </span>CMD
<span class="w"> </span><span class="m">8876</span><span class="w"> </span>-<span class="w"> </span>./server
<span class="w"> </span>-<span class="w"> </span><span class="m">8876</span><span class="w"> </span>-
<span class="w"> </span>-<span class="w"> </span><span class="m">8877</span><span class="w"> </span>-
<span class="w"> </span>-<span class="w"> </span><span class="m">8878</span><span class="w"> </span>-
<span class="w"> </span>-<span class="w"> </span><span class="m">8879</span><span class="w"> </span>-
<span class="w"> </span>-<span class="w"> </span><span class="m">8880</span><span class="w"> </span>-
</code></pre></div>
<p>We have five threads here. One is our main thread and the other four are the threadpool threads. The <code>m</code> option tells ps to show the threads of a process. The <code>-o</code> option specifies fields to output. We can even get fancy and output the addresses of the threads' stack pointers and instruction pointers, like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ps<span class="w"> </span>m<span class="w"> </span>--pid<span class="w"> </span><span class="m">8876</span><span class="w"> </span>-o<span class="w"> </span>pid,tid,cmd,esp,eip
<span class="w"> </span>PID<span class="w"> </span>TID<span class="w"> </span>CMD<span class="w"> </span>ESP<span class="w"> </span>EIP
<span class="w"> </span><span class="m">8876</span><span class="w"> </span>-<span class="w"> </span>./server<span class="w"> </span>-<span class="w"> </span>-
<span class="w"> </span>-<span class="w"> </span><span class="m">8876</span><span class="w"> </span>-<span class="w"> </span>a0880b70<span class="w"> </span>ccb0e7ad
<span class="w"> </span>-<span class="w"> </span><span class="m">8877</span><span class="w"> </span>-<span class="w"> </span>cc733ec0<span class="w"> </span>ccb0b3a0
<span class="w"> </span>-<span class="w"> </span><span class="m">8878</span><span class="w"> </span>-<span class="w"> </span>cbf32ec0<span class="w"> </span>ccb0b3a0
<span class="w"> </span>-<span class="w"> </span><span class="m">8879</span><span class="w"> </span>-<span class="w"> </span>cb731ec0<span class="w"> </span>ccb0b3a0
<span class="w"> </span>-<span class="w"> </span><span class="m">8880</span><span class="w"> </span>-<span class="w"> </span>caf30ec0<span class="w"> </span>ccb0b3a0
</code></pre></div>
<p>So all the threads are at the same instruction, but they have different stack pointers, which makes sense. If I execute something on one of the threads, both the <code>ESP</code> and <code>EIP</code> can possibly change.</p>
<p>A lot of data about processes lives in the <code>proc</code> filesytem, located in <code>/proc</code>. For each running process, there's a subdirectory of <code>/proc</code> named after the process id. For example, for our process <code>8876</code>, there's a <code>status</code> file which lists various information about the process. Let's look at it:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>/proc/8876/status
Name:<span class="w"> </span>server
State:<span class="w"> </span>S<span class="w"> </span><span class="o">(</span>sleeping<span class="o">)</span>
Tgid:<span class="w"> </span><span class="m">8876</span>
Ngid:<span class="w"> </span><span class="m">0</span>
Pid:<span class="w"> </span><span class="m">8876</span>
PPid:<span class="w"> </span><span class="m">2604</span>
TracerPid:<span class="w"> </span><span class="m">0</span>
Uid:<span class="w"> </span><span class="m">1000</span><span class="w"> </span><span class="m">1000</span><span class="w"> </span><span class="m">1000</span><span class="w"> </span><span class="m">1000</span>
Gid:<span class="w"> </span><span class="m">1000</span><span class="w"> </span><span class="m">1000</span><span class="w"> </span><span class="m">1000</span><span class="w"> </span><span class="m">1000</span>
FDSize:<span class="w"> </span><span class="m">256</span>
Groups:<span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="m">24</span><span class="w"> </span><span class="m">27</span><span class="w"> </span><span class="m">30</span><span class="w"> </span><span class="m">46</span><span class="w"> </span><span class="m">113</span><span class="w"> </span><span class="m">128</span><span class="w"> </span><span class="m">999</span><span class="w"> </span><span class="m">1000</span><span class="w"> </span>
NStgid:<span class="w"> </span><span class="m">8876</span>
NSpid:<span class="w"> </span><span class="m">8876</span>
NSpgid:<span class="w"> </span><span class="m">8876</span>
NSsid:<span class="w"> </span><span class="m">2604</span>
VmPeak:<span class="w"> </span><span class="m">39316</span><span class="w"> </span>kB
VmSize:<span class="w"> </span><span class="m">39316</span><span class="w"> </span>kB
VmLck:<span class="w"> </span><span class="m">0</span><span class="w"> </span>kB
VmPin:<span class="w"> </span><span class="m">0</span><span class="w"> </span>kB
VmHWM:<span class="w"> </span><span class="m">800</span><span class="w"> </span>kB
VmRSS:<span class="w"> </span><span class="m">800</span><span class="w"> </span>kB
VmData:<span class="w"> </span><span class="m">32988</span><span class="w"> </span>kB
VmStk:<span class="w"> </span><span class="m">136</span><span class="w"> </span>kB
VmExe:<span class="w"> </span><span class="m">16</span><span class="w"> </span>kB
VmLib:<span class="w"> </span><span class="m">2040</span><span class="w"> </span>kB
VmPTE:<span class="w"> </span><span class="m">48</span><span class="w"> </span>kB
VmPMD:<span class="w"> </span><span class="m">12</span><span class="w"> </span>kB
VmSwap:<span class="w"> </span><span class="m">0</span><span class="w"> </span>kB
HugetlbPages:<span class="w"> </span><span class="m">0</span><span class="w"> </span>kB
Threads:<span class="w"> </span><span class="m">5</span>
SigQ:<span class="w"> </span><span class="m">0</span>/7848
SigPnd:<span class="w"> </span><span class="m">0000000000000000</span>
ShdPnd:<span class="w"> </span><span class="m">0000000000000000</span>
SigBlk:<span class="w"> </span><span class="m">0000000000000000</span>
SigIgn:<span class="w"> </span><span class="m">0000000000000000</span>
SigCgt:<span class="w"> </span><span class="m">0000000180000000</span>
CapInh:<span class="w"> </span><span class="m">0000000000000000</span>
CapPrm:<span class="w"> </span><span class="m">0000000000000000</span>
CapEff:<span class="w"> </span><span class="m">0000000000000000</span>
CapBnd:<span class="w"> </span>0000003fffffffff
CapAmb:<span class="w"> </span><span class="m">0000000000000000</span>
Seccomp:<span class="w"> </span><span class="m">0</span>
Cpus_allowed:<span class="w"> </span><span class="m">1</span>
Cpus_allowed_list:<span class="w"> </span><span class="m">0</span>
Mems_allowed:<span class="w"> </span><span class="m">00000000</span>,00000001
Mems_allowed_list:<span class="w"> </span><span class="m">0</span>
voluntary_ctxt_switches:<span class="w"> </span><span class="m">3</span>
nonvoluntary_ctxt_switches:<span class="w"> </span><span class="m">2</span>
</code></pre></div>
<p>There's a lot of data in here, but remember how we used <code>ps</code> to count the number of threads in this process. That's also available here on the line saying <code>Threads: 5</code>.</p>
<p>Our last command is <code>pidstat</code>. <code>pidstat</code> shows statistics about a running process, which can be updated at a regular time interval. A possible invocation can be:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pidstat<span class="w"> </span>-p<span class="w"> </span><span class="m">8876</span><span class="w"> </span><span class="m">1</span>
Linux<span class="w"> </span><span class="m">4</span>.4.0-64-generic<span class="w"> </span><span class="o">(</span>virtbox<span class="o">)</span><span class="w"> </span><span class="m">03</span>/01/2017<span class="w"> </span>_x86_64_<span class="w"> </span><span class="o">(</span><span class="m">1</span><span class="w"> </span>CPU<span class="o">)</span>
<span class="m">12</span>:22:00<span class="w"> </span>PM<span class="w"> </span>UID<span class="w"> </span>PID<span class="w"> </span>%usr<span class="w"> </span>%system<span class="w"> </span>%guest<span class="w"> </span>%CPU<span class="w"> </span>CPU<span class="w"> </span>Command
<span class="m">12</span>:22:01<span class="w"> </span>PM<span class="w"> </span><span class="m">1000</span><span class="w"> </span><span class="m">8876</span><span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span><span class="w"> </span>server
<span class="m">12</span>:22:02<span class="w"> </span>PM<span class="w"> </span><span class="m">1000</span><span class="w"> </span><span class="m">8876</span><span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span><span class="w"> </span>server
</code></pre></div>
<p>Our server is not doing anything right now, so you see a lot of zeroes.</p>
<p>There are many other interesting commands that you can look to figure out what processes are doing. <code>strace</code> shows system calls run by a process. <code>ltrace</code> shows dynamic library calls. <code>tcpdump</code> can be used to show traffic going in and out of a process.</p>
<p>So, that's all for today. Happy running of processes.</p>The Linux init system.2017-02-13T21:09:00-08:002017-02-13T21:09:00-08:00Petko Minkovtag:pminkov.github.io,2017-02-13:/blog/the-linux-init-system.html<p>I decided that I'll dig down into <a href="https://en.wikipedia.org/wiki/Init">init systems</a> in Linux and learn more about them. I'm running Ubuntu 16.04, so this might look different on other distributions.</p>
<p>The init system in Linux is mainly responsible for starting essential service processes, mounting file systems and possibly other tasks. The …</p><p>I decided that I'll dig down into <a href="https://en.wikipedia.org/wiki/Init">init systems</a> in Linux and learn more about them. I'm running Ubuntu 16.04, so this might look different on other distributions.</p>
<p>The init system in Linux is mainly responsible for starting essential service processes, mounting file systems and possibly other tasks. The main init systems are systemd, System V init and Upstart. Ubuntu uses systemd.</p>
<p>The init system starts after the Kernel starts its first user space process - init. Indeed, let's see what's running with PID 1:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ps<span class="w"> </span><span class="m">1</span>
<span class="w"> </span>PID<span class="w"> </span>TTY<span class="w"> </span>STAT<span class="w"> </span>TIME<span class="w"> </span>COMMAND
<span class="w"> </span><span class="m">1</span><span class="w"> </span>?<span class="w"> </span>Ss<span class="w"> </span><span class="m">0</span>:04<span class="w"> </span>/sbin/init<span class="w"> </span>splash
</code></pre></div>
<p>It's <code>/sbin/init</code>. Let's see what this file is:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-l<span class="w"> </span>/sbin/init
lrwxrwxrwx<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">20</span><span class="w"> </span>Sep<span class="w"> </span><span class="m">28</span><span class="w"> </span><span class="m">18</span>:40<span class="w"> </span>/sbin/init<span class="w"> </span>-><span class="w"> </span>/lib/systemd/systemd
</code></pre></div>
<p>From this output, we can figure out that Ubuntu is using systemd. systemd is a fairly new project (initial release was 6 years ago), but it looks like its widely adopted now. systemd would take care of running various services like your ssh server, your web server and various other ones which are more "under the hood" oriented.</p>
<p>systemd organizes itself with unit files which contain the description of various units and their dependencies. The units are organized in configuration files, which live in various directories. The main directories are:</p>
<ul>
<li>System unit directory: <code>/usr/lib/systemd/</code>. Your distribution maintains this, so don't edit it.</li>
<li>System configuration directory: <code>/etc/systemd</code>. Make your local changes here.</li>
</ul>
<p>Well, these are not all directories that contain unit files. Here's the full set of paths that systemd uses:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>systemctl<span class="w"> </span>-p<span class="w"> </span>UnitPath<span class="w"> </span>show
<span class="nv">UnitPath</span><span class="o">=</span>/etc/systemd/system<span class="w"> </span>/run/systemd/system<span class="w"> </span>/run/systemd/generator<span class="w"> </span>/usr/local/lib/systemd/system<span class="w"> </span>/lib/systemd/system<span class="w"> </span>/usr/lib/systemd/system<span class="w"> </span>/run/systemd/generator.late
</code></pre></div>
<p>I won't go into details about what the unit files contain, but instead look at two services that I was curious about - ssh and apache. Who runs them? When are they run? How can I verify that they are running?</p>
<p>Let's start with ssh. The main command to interface with systemd is <code>systemctl</code>. We can use it to list all services that are running, by calling <code>systemctl list-units</code>. Let's look for ssh in here:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>systemctl<span class="w"> </span>list-units<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>ssh
ssh.service<span class="w"> </span>loaded<span class="w"> </span>active<span class="w"> </span>running<span class="w"> </span>OpenBSD<span class="w"> </span>Secure<span class="w"> </span>Shell<span class="w"> </span>server
</code></pre></div>
<p>Indeed, we have ssh running. Now, let's look at its status and its config file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>systemctl<span class="w"> </span>status<span class="w"> </span>ssh
●<span class="w"> </span>ssh.service<span class="w"> </span>-<span class="w"> </span>OpenBSD<span class="w"> </span>Secure<span class="w"> </span>Shell<span class="w"> </span>server
<span class="w"> </span>Loaded:<span class="w"> </span>loaded<span class="w"> </span><span class="o">(</span>/lib/systemd/system/ssh.service<span class="p">;</span><span class="w"> </span>enabled<span class="p">;</span><span class="w"> </span>vendor<span class="w"> </span>preset:<span class="w"> </span>enabled<span class="o">)</span>
<span class="w"> </span>Active:<span class="w"> </span>active<span class="w"> </span><span class="o">(</span>running<span class="o">)</span><span class="w"> </span>since<span class="w"> </span>Thu<span class="w"> </span><span class="m">2017</span>-02-09<span class="w"> </span><span class="m">19</span>:27:57<span class="w"> </span>PST<span class="p">;</span><span class="w"> </span><span class="m">4</span><span class="w"> </span>days<span class="w"> </span>ago
<span class="w"> </span>Main<span class="w"> </span>PID:<span class="w"> </span><span class="m">786</span><span class="w"> </span><span class="o">(</span>sshd<span class="o">)</span>
<span class="w"> </span>Tasks:<span class="w"> </span><span class="m">1</span>
<span class="w"> </span>Memory:<span class="w"> </span><span class="m">6</span>.4M
<span class="w"> </span>CPU:<span class="w"> </span>199ms
<span class="w"> </span>CGroup:<span class="w"> </span>/system.slice/ssh.service
<span class="w"> </span>└─786<span class="w"> </span>/usr/sbin/sshd<span class="w"> </span>-D
Feb<span class="w"> </span><span class="m">12</span><span class="w"> </span><span class="m">18</span>:23:50<span class="w"> </span>virtbox<span class="w"> </span>sshd<span class="o">[</span><span class="m">5791</span><span class="o">]</span>:<span class="w"> </span>Accepted<span class="w"> </span>password<span class="w"> </span><span class="k">for</span><span class="w"> </span>petko<span class="w"> </span>from<span class="w"> </span><span class="m">192</span>.168.1.86<span class="w"> </span>port<span class="w"> </span><span class="m">57805</span><span class="w"> </span>ssh2
Feb<span class="w"> </span><span class="m">12</span><span class="w"> </span><span class="m">18</span>:23:50<span class="w"> </span>virtbox<span class="w"> </span>sshd<span class="o">[</span><span class="m">5791</span><span class="o">]</span>:<span class="w"> </span>pam_unix<span class="o">(</span>sshd:session<span class="o">)</span>:<span class="w"> </span>session<span class="w"> </span>opened<span class="w"> </span><span class="k">for</span><span class="w"> </span>user<span class="w"> </span>petko<span class="w"> </span>by<span class="w"> </span><span class="o">(</span><span class="nv">uid</span><span class="o">=</span><span class="m">0</span><span class="o">)</span>
...
</code></pre></div>
<p>So here it is. The ssh service is running as process 786. We can see that this process is listening on port 22:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>netstat<span class="w"> </span>-tulpn<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="m">786</span>
tcp<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>.0.0.0:22<span class="w"> </span><span class="m">0</span>.0.0.0:*<span class="w"> </span>LISTEN<span class="w"> </span><span class="m">786</span>/sshd<span class="w"> </span>
tcp6<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span>:::22<span class="w"> </span>:::*<span class="w"> </span>LISTEN<span class="w"> </span><span class="m">786</span>/sshd<span class="w"> </span>
</code></pre></div>
<p>Indeed, it is. <code>systemctl</code> has another useful command that allows us to print the configuration file for a unit. It works like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>systemctl<span class="w"> </span>cat<span class="w"> </span>ssh
<span class="c1"># /lib/systemd/system/ssh.service</span>
<span class="o">[</span>Unit<span class="o">]</span>
<span class="nv">Description</span><span class="o">=</span>OpenBSD<span class="w"> </span>Secure<span class="w"> </span>Shell<span class="w"> </span>server
<span class="nv">After</span><span class="o">=</span>network.target<span class="w"> </span>auditd.service
<span class="nv">ConditionPathExists</span><span class="o">=</span>!/etc/ssh/sshd_not_to_be_run
<span class="o">[</span>Service<span class="o">]</span>
<span class="nv">EnvironmentFile</span><span class="o">=</span>-/etc/default/ssh
<span class="nv">ExecStart</span><span class="o">=</span>/usr/sbin/sshd<span class="w"> </span>-D<span class="w"> </span><span class="nv">$SSHD_OPTS</span>
<span class="nv">ExecReload</span><span class="o">=</span>/bin/kill<span class="w"> </span>-HUP<span class="w"> </span><span class="nv">$MAINPID</span>
<span class="nv">KillMode</span><span class="o">=</span>process
<span class="nv">Restart</span><span class="o">=</span>on-failure
<span class="nv">RestartPreventExitStatus</span><span class="o">=</span><span class="m">255</span>
<span class="nv">Type</span><span class="o">=</span>notify
<span class="o">[</span>Install<span class="o">]</span>
<span class="nv">WantedBy</span><span class="o">=</span>multi-user.target
<span class="nv">Alias</span><span class="o">=</span>sshd.service
</code></pre></div>
<p>So here you can see where is the configuration file located.</p>
<p>Alright, enough ssh. Let's move on to apache. First, a little history though. Before systemd, apparently the main init system in Linux was System V. System V is different than systemd, because it executes services in sequential order, while systemd can be parallel. System V also can't start services on "as-needed" basis. So I guess that's why systemd was implemented. systemd has its config files in <code>/etc/init.d</code>. That's where Apache installs its command files as well - it doesn't create systemd unit files. However, systemd knows how to execute the System V init files. I won't go into details of how System V init works, but basically it executes commands on different runlevels and at each runlevel they are executed in sequential order.</p>
<p>Let's see how apache looks like in systemd:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>systemctl<span class="w"> </span>list-units<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>apache
apache2.service<span class="w"> </span>loaded<span class="w"> </span>active<span class="w"> </span>running<span class="w"> </span>LSB:<span class="w"> </span>Apache2<span class="w"> </span>web<span class="w"> </span>server
</code></pre></div>
<p>It's running. Now let's get its status:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>systemctl<span class="w"> </span>status<span class="w"> </span>apache2
●<span class="w"> </span>apache2.service<span class="w"> </span>-<span class="w"> </span>LSB:<span class="w"> </span>Apache2<span class="w"> </span>web<span class="w"> </span>server
<span class="w"> </span>Loaded:<span class="w"> </span>loaded<span class="w"> </span><span class="o">(</span>/etc/init.d/apache2<span class="p">;</span><span class="w"> </span>bad<span class="p">;</span><span class="w"> </span>vendor<span class="w"> </span>preset:<span class="w"> </span>enabled<span class="o">)</span>
<span class="w"> </span>Drop-In:<span class="w"> </span>/lib/systemd/system/apache2.service.d
<span class="w"> </span>└─apache2-systemd.conf
<span class="w"> </span>Active:<span class="w"> </span>active<span class="w"> </span><span class="o">(</span>running<span class="o">)</span><span class="w"> </span>since<span class="w"> </span>Mon<span class="w"> </span><span class="m">2017</span>-02-13<span class="w"> </span><span class="m">12</span>:49:55<span class="w"> </span>PST<span class="p">;</span><span class="w"> </span>8h<span class="w"> </span>ago
<span class="w"> </span>Docs:<span class="w"> </span>man:systemd-sysv-generator<span class="o">(</span><span class="m">8</span><span class="o">)</span>
<span class="w"> </span>Tasks:<span class="w"> </span><span class="m">55</span>
<span class="w"> </span>Memory:<span class="w"> </span><span class="m">6</span>.5M
<span class="w"> </span>CPU:<span class="w"> </span><span class="m">17</span>.042s
<span class="w"> </span>CGroup:<span class="w"> </span>/system.slice/apache2.service
<span class="w"> </span>├─9097<span class="w"> </span>/usr/sbin/apache2<span class="w"> </span>-k<span class="w"> </span>start
<span class="w"> </span>├─9100<span class="w"> </span>/usr/sbin/apache2<span class="w"> </span>-k<span class="w"> </span>start
<span class="w"> </span>└─9101<span class="w"> </span>/usr/sbin/apache2<span class="w"> </span>-k<span class="w"> </span>start
Feb<span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="m">12</span>:49:54<span class="w"> </span>virtbox<span class="w"> </span>systemd<span class="o">[</span><span class="m">1</span><span class="o">]</span>:<span class="w"> </span>Starting<span class="w"> </span>LSB:<span class="w"> </span>Apache2<span class="w"> </span>web<span class="w"> </span>server...
Feb<span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="m">12</span>:49:54<span class="w"> </span>virtbox<span class="w"> </span>apache2<span class="o">[</span><span class="m">9071</span><span class="o">]</span>:<span class="w"> </span>*<span class="w"> </span>Starting<span class="w"> </span>Apache<span class="w"> </span>httpd<span class="w"> </span>web<span class="w"> </span>server<span class="w"> </span>apache2
Feb<span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="m">12</span>:49:54<span class="w"> </span>virtbox<span class="w"> </span>apache2<span class="o">[</span><span class="m">9071</span><span class="o">]</span>:<span class="w"> </span>AH00558:<span class="w"> </span>apache2:<span class="w"> </span>Could<span class="w"> </span>not<span class="w"> </span>reliably<span class="w"> </span>determine<span class="w"> </span>the<span class="w"> </span>server<span class="s1">'s fully qualified domain name, using 127.0.1.1. Set the '</span>ServerName<span class="err">'</span><span class="w"> </span>directive<span class="w"> </span>globall
Feb<span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="m">12</span>:49:55<span class="w"> </span>virtbox<span class="w"> </span>apache2<span class="o">[</span><span class="m">9071</span><span class="o">]</span>:<span class="w"> </span>*
Feb<span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="m">12</span>:49:55<span class="w"> </span>virtbox<span class="w"> </span>systemd<span class="o">[</span><span class="m">1</span><span class="o">]</span>:<span class="w"> </span>Started<span class="w"> </span>LSB:<span class="w"> </span>Apache2<span class="w"> </span>web<span class="w"> </span>server.
</code></pre></div>
<p>Look at something interesting here. The file responsible for starting apache is listed as <code>/etc/init.d/apache2</code>. That's the file indeed. It's adapted into systemd by using the <code>systemd-sysv-generator</code>. So that is systemd running. We can run a cat to see that:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>systemctl<span class="w"> </span>cat<span class="w"> </span>apache2
<span class="c1"># /run/systemd/generator.late/apache2.service</span>
<span class="c1"># Automatically generated by systemd-sysv-generator</span>
<span class="o">[</span>Unit<span class="o">]</span>
<span class="nv">Documentation</span><span class="o">=</span>man:systemd-sysv-generator<span class="o">(</span><span class="m">8</span><span class="o">)</span>
<span class="nv">SourcePath</span><span class="o">=</span>/etc/init.d/apache2
<span class="nv">Description</span><span class="o">=</span>LSB:<span class="w"> </span>Apache2<span class="w"> </span>web<span class="w"> </span>server
<span class="nv">Before</span><span class="o">=</span>multi-user.target
<span class="nv">Before</span><span class="o">=</span>multi-user.target
<span class="nv">Before</span><span class="o">=</span>multi-user.target
<span class="nv">Before</span><span class="o">=</span>graphical.target
<span class="nv">Before</span><span class="o">=</span>shutdown.target
<span class="nv">After</span><span class="o">=</span>local-fs.target
<span class="nv">After</span><span class="o">=</span>remote-fs.target
<span class="nv">After</span><span class="o">=</span>network-online.target
<span class="nv">After</span><span class="o">=</span>systemd-journald-dev-log.socket
<span class="nv">After</span><span class="o">=</span>nss-lookup.target
<span class="nv">Wants</span><span class="o">=</span>network-online.target
<span class="nv">Conflicts</span><span class="o">=</span>shutdown.target
<span class="o">[</span>Service<span class="o">]</span>
<span class="nv">Type</span><span class="o">=</span>forking
<span class="nv">Restart</span><span class="o">=</span>no
<span class="nv">TimeoutSec</span><span class="o">=</span>5min
<span class="nv">IgnoreSIGPIPE</span><span class="o">=</span>no
<span class="nv">KillMode</span><span class="o">=</span>process
<span class="nv">GuessMainPID</span><span class="o">=</span>no
<span class="nv">RemainAfterExit</span><span class="o">=</span>yes
<span class="nv">ExecStart</span><span class="o">=</span>/etc/init.d/apache2<span class="w"> </span>start
<span class="nv">ExecStop</span><span class="o">=</span>/etc/init.d/apache2<span class="w"> </span>stop
<span class="nv">ExecReload</span><span class="o">=</span>/etc/init.d/apache2<span class="w"> </span>reload
<span class="c1"># /lib/systemd/system/apache2.service.d/apache2-systemd.conf</span>
<span class="o">[</span>Service<span class="o">]</span>
<span class="nv">Type</span><span class="o">=</span>forking
<span class="nv">RemainAfterExit</span><span class="o">=</span>no
</code></pre></div>
<p>This is the file that systemd created in order to integrate the System V init command into its system.</p>
<p>What are some other interesting systemctl commands? Let's list them:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Start / stop /restart a service.</span>
$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span>restart<span class="w"> </span>apache2
<span class="c1"># List all services:</span>
$<span class="w"> </span>systemctl<span class="w"> </span>list-units<span class="w"> </span>--type<span class="o">=</span>service
<span class="c1"># List dependencies:</span>
$<span class="w"> </span>systemctl<span class="w"> </span>list-dependencies<span class="w"> </span>sshd.service
<span class="c1"># See low level properties of a unit:</span>
$<span class="w"> </span>systemctl<span class="w"> </span>show<span class="w"> </span>sshd.service
</code></pre></div>
<p>And one last cool command:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>systemd-analyze
Startup<span class="w"> </span>finished<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">3</span>.989s<span class="w"> </span><span class="o">(</span>kernel<span class="o">)</span><span class="w"> </span>+<span class="w"> </span><span class="m">7</span>.673s<span class="w"> </span><span class="o">(</span>userspace<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">11</span>.663s
</code></pre></div>
<p>This command prints the time it took to startup our system.</p>
<p>So that's all for today. There's definitely more to explore in systemd land - the syntax of unit files, how systemd executes them and so on. I'll leave that for some other time.</p>How to shut down and restore an Elastic Beanstalk environment.2017-01-30T12:07:00-08:002017-01-30T12:07:00-08:00Petko Minkovtag:pminkov.github.io,2017-01-30:/blog/how-to-shut-down-and-restore-an-elastic-beanstalk-environment.html<p>Let's say you're running an <a href="https://aws.amazon.com/elasticbeanstalk/">Elastic Beanstalk</a> application. You might want to stop it so that you're not paying money for it. There's one way to do this by running commands. You can use <code>eb terminate</code> and <code>eb restore</code>, but if you terminate a setup with a database and you …</p><p>Let's say you're running an <a href="https://aws.amazon.com/elasticbeanstalk/">Elastic Beanstalk</a> application. You might want to stop it so that you're not paying money for it. There's one way to do this by running commands. You can use <code>eb terminate</code> and <code>eb restore</code>, but if you terminate a setup with a database and you restore it, the contents of the database won't be restored. You can also only restore an environment that has been terminated within the last 6 weeks.</p>
<p>Let's see how we can terminate and restore without the 6 weeks restriction and let's also see how does the database backup and restore look like.</p>
<p>The first thing that you have to do is to save your environment, which pretty much consists of:</p>
<ul>
<li>Elastic Beanstalk <a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environment-configuration-savedconfig.html">configuration</a>. This is located in the <code>.elasticbeanstalk</code> directory.</li>
<li><a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html">Environment customizations</a>. They're located in the <code>.ebextensions</code> directory.</li>
<li>Your source code. Located in your base directory.</li>
<li>A database (optional).</li>
</ul>
<p>My particular app is a single instance Python/Django application with a MySQL database hosted in RDS.</p>
<p>So let's get started.</p>
<h2 id="backing-up-your-environment">Backing up your environment.</h2>
<p>First, get the name of your environment. Mine is <code>neatlinks-dev</code>. Run the following commands after that:</p>
<div class="highlight"><pre><span></span><code><span class="nb">export</span><span class="w"> </span><span class="nv">EB_DATE_LABEL</span><span class="o">=</span><span class="sb">`</span>date<span class="w"> </span>+<span class="s2">"D%F-T%H-%M-%S"</span><span class="sb">`</span>
<span class="c1"># Name of saved config.</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">EB_CONFIG</span><span class="o">=</span>saved-<span class="nv">$EB_DATE_LABEL</span>
<span class="c1"># Name of database snapshot.</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">EB_SNAPSHOT_NAME</span><span class="o">=</span>snapshot-<span class="nv">$EB_DATE_LABEL</span>
<span class="c1"># Instance id for our current database.</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">EB_DB</span><span class="o">=</span><span class="s2">"aaeag9ndvxonft"</span>
<span class="c1"># Save the environment cname, we'll need it later.</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">EB_CNAME</span><span class="o">=</span><span class="s1">'neatlinks-dev-www'</span>
<span class="c1"># Save current config.</span>
eb<span class="w"> </span>config<span class="w"> </span>save<span class="w"> </span>neatlinks-dev<span class="w"> </span>--cfg<span class="w"> </span><span class="nv">$EB_CONFIG</span>
<span class="c1"># Create and wait for database snapshot.</span>
aws<span class="w"> </span>rds<span class="w"> </span>create-db-snapshot<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--db-instance-identifier<span class="w"> </span><span class="nv">$EB_DB</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--db-snapshot-identifier<span class="w"> </span><span class="nv">$EB_SNAPSHOT_NAME</span>
<span class="nb">time</span><span class="w"> </span>aws<span class="w"> </span>rds<span class="w"> </span><span class="nb">wait</span><span class="w"> </span>db-snapshot-completed<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--db-instance-identifier<span class="w"> </span><span class="nv">$EB_DB</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--db-snapshot-identifier<span class="w"> </span><span class="nv">$EB_SNAPSHOT_NAME</span>
</code></pre></div>
<p>So we did two things:</p>
<ol>
<li>We saved the environment configuration. You can list your saved configurations by running <code>eb config list</code>.</li>
<li>We created a snapshot of the database.</li>
</ol>
<p>At this point we can terminate our environment by calling <code>eb terminate</code>. And that's it - Amazon is not charging you anymore - your EC2 instance, RDS instance, load balancers, etc. are down. I don't have a load balancer, but I can verify that everything else is gone.</p>
<p>You should now commit your configs to your source control so that you have them saved when you come back.</p>
<p>Now, your application is sitting safely saved for a while and you decide to restore it.</p>
<h2 id="restoring-your-environment">Restoring your environment.</h2>
<p>If you have deleted your application, you'll need to run <code>eb init</code>. After that, we'll run the following commands:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Create environment. It's important to specify a cname, so that we don't have to</span>
<span class="c1"># change our DNS config. This takes a lot of time, so I bump the timeout.</span>
<span class="c1"># This command will print the name of your new database instance.</span>
<span class="nb">time</span><span class="w"> </span>eb<span class="w"> </span>create<span class="w"> </span>neatlinks-dev<span class="w"> </span>--cfg<span class="w"> </span><span class="nv">$EB_CONFIG</span><span class="w"> </span>--cname<span class="w"> </span><span class="nv">$EB_CNAME</span><span class="w"> </span>--timeout<span class="w"> </span><span class="m">30</span>
<span class="c1"># Get this id from the printed output of "eb create".</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">EB_NEW_DB</span><span class="o">=</span><span class="s2">"aa1q3no37rzd407"</span>
<span class="c1"># Delete DB instance. We'll replace it with the snapshot.</span>
aws<span class="w"> </span>rds<span class="w"> </span>delete-db-instance<span class="w"> </span>--db-instance-identifier<span class="w"> </span><span class="nv">$EB_NEW_DB</span><span class="w"> </span>--skip-final-snapshot
aws<span class="w"> </span>rds<span class="w"> </span><span class="nb">wait</span><span class="w"> </span>db-instance-deleted<span class="w"> </span>--db-instance-identifier<span class="w"> </span><span class="nv">$EB_NEW_DB</span>
<span class="c1"># Restore from snapshot.</span>
aws<span class="w"> </span>rds<span class="w"> </span>restore-db-instance-from-db-snapshot<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--db-instance-identifier<span class="w"> </span><span class="nv">$EB_NEW_DB</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--db-snapshot-identifier<span class="w"> </span><span class="nv">$EB_SNAPSHOT_NAME</span>
aws<span class="w"> </span>rds<span class="w"> </span><span class="nb">wait</span><span class="w"> </span>db-instance-available<span class="w"> </span>--db-instance-identifier<span class="w"> </span><span class="nv">$EB_NEW_DB</span>
</code></pre></div>
<p>Your production setup is ready now! Your application probably won't work though, because your DB endpoint is different. Edit your source code or your own configuration files so that your application connects to the new database. Once done, call <code>eb deploy</code>.</p>
<p>That's it. Your application should be running now. You can use this procedure to rename your Elastic Beanstalk application and environment as well, which is a feature that's missing from the <code>eb</code> CLI app.</p>What happens when you run out of memory in Linux?2017-01-15T12:50:00-08:002017-01-15T12:50:00-08:00Petko Minkovtag:pminkov.github.io,2017-01-15:/blog/what-happens-when-you-run-out-of-memory-in-linux.html<p>I've always been curious to figure out what happens when you run out of memory in Linux and recently I was experimenting with something that helped me figure it out. </p>
<p>I was trying out <a href="http://dhbox.org/">dhbox</a> deployment on an EC2 machine. dhbox allows you to start a virtual environment in which …</p><p>I've always been curious to figure out what happens when you run out of memory in Linux and recently I was experimenting with something that helped me figure it out. </p>
<p>I was trying out <a href="http://dhbox.org/">dhbox</a> deployment on an EC2 machine. dhbox allows you to start a virtual environment in which you can try out various data science tools. These virtual environments are run on Docker containers. These Docker containers take a lot of memory, so you can't run too many on a single machine. So far so good. But let's see what actually happens.</p>
<p>Before we start, <a href="http://techblog.netflix.com/2015/11/linux-performance-analysis-in-60s.html">here</a> is a great document from the Netflix Eng blog that describes the tools that can be used to debug a slow Linux box. </p>
<p>I deployed DHBox on an Ubuntu image in EC2 with 1GB of memory and no swap file. Yes, no swap file. Why? It seems like that's done, because having a swap file might incur a lot of EBS IO, which leads to high pricing. But still, this makes for an interesting debugging scenario, so let's continue.</p>
<p>Here's what our free memory situation is in the beginnging:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>free<span class="w"> </span>-m
<span class="w"> </span>total<span class="w"> </span>used<span class="w"> </span>free<span class="w"> </span>shared<span class="w"> </span>buff/cache<span class="w"> </span>available
Mem:<span class="w"> </span><span class="m">990</span><span class="w"> </span><span class="m">78</span><span class="w"> </span><span class="m">518</span><span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="m">392</span><span class="w"> </span><span class="m">867</span>
Swap:<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
</code></pre></div>
<p>Notice how we have 392MB in "buff/cache". What is this? <a href="http://www.tldp.org/LDP/sag/html/buffer-cache.html">Here</a> is a good explanation of it. The buffer cache is caching (in RAM) data that's on disk. For example, the "ls" command, or the glibc library are things that are often used and are good candidates for caching.</p>
<p>The first thing I do is to start the Python web app for DHBox. It's a simple Flask app running on <a href="http://gunicorn.org/">gunicorn</a>. After I start it, this is what <code>free</code> is showing:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>free<span class="w"> </span>-m
<span class="w"> </span>total<span class="w"> </span>used<span class="w"> </span>free<span class="w"> </span>shared<span class="w"> </span>buff/cache<span class="w"> </span>available
Mem:<span class="w"> </span><span class="m">990</span><span class="w"> </span><span class="m">127</span><span class="w"> </span><span class="m">453</span><span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="m">410</span><span class="w"> </span><span class="m">818</span>
Swap:<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
</code></pre></div>
<p>Reasonable. We are using a bit more memory, we have cached a bit more data from disk.</p>
<p>Now, I'll start running vmstat and run four virtual labs. This seems to be enough to take 1GB of memory and pretty much bring down the machine. One other tool we're going to use is <code>vmstat</code>. <code>vmstat</code> is great, it's outputting a lot of useful information. Let's see how it looks like before we start running the Docker containers:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>vmstat<span class="w"> </span><span class="m">5</span>
procs<span class="w"> </span>-----------memory----------<span class="w"> </span>---swap--<span class="w"> </span>-----io----<span class="w"> </span>-system--<span class="w"> </span>------cpu-----
<span class="w"> </span>r<span class="w"> </span>b<span class="w"> </span>swpd<span class="w"> </span>free<span class="w"> </span>buff<span class="w"> </span>cache<span class="w"> </span>si<span class="w"> </span>so<span class="w"> </span>bi<span class="w"> </span>bo<span class="w"> </span><span class="k">in</span><span class="w"> </span>cs<span class="w"> </span>us<span class="w"> </span>sy<span class="w"> </span>id<span class="w"> </span>wa<span class="w"> </span>st
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">463880</span><span class="w"> </span><span class="m">51328</span><span class="w"> </span><span class="m">369012</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">635</span><span class="w"> </span><span class="m">32</span><span class="w"> </span><span class="m">94</span><span class="w"> </span><span class="m">231</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">95</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">463356</span><span class="w"> </span><span class="m">51336</span><span class="w"> </span><span class="m">369044</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">36</span><span class="w"> </span><span class="m">68</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">100</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">462884</span><span class="w"> </span><span class="m">51372</span><span class="w"> </span><span class="m">369116</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">14</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">46</span><span class="w"> </span><span class="m">69</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">99</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">462380</span><span class="w"> </span><span class="m">51372</span><span class="w"> </span><span class="m">369116</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">101</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">100</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
</code></pre></div>
<p>The telling part here is the <code>id</code> column. That's the percentage which the CPU spends being idle. As you can see, the CPU is pretty idle. Not much to do.</p>
<p>We're then starting the first Docker container. Things spike up for a while. Let's observe vmstat:</p>
<div class="highlight"><pre><span></span><code>procs<span class="w"> </span>-----------memory----------<span class="w"> </span>---swap--<span class="w"> </span>-----io----<span class="w"> </span>-system--<span class="w"> </span>------cpu-----
<span class="w"> </span>r<span class="w"> </span>b<span class="w"> </span>swpd<span class="w"> </span>free<span class="w"> </span>buff<span class="w"> </span>cache<span class="w"> </span>si<span class="w"> </span>so<span class="w"> </span>bi<span class="w"> </span>bo<span class="w"> </span><span class="k">in</span><span class="w"> </span>cs<span class="w"> </span>us<span class="w"> </span>sy<span class="w"> </span>id<span class="w"> </span>wa<span class="w"> </span>st
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">462380</span><span class="w"> </span><span class="m">51372</span><span class="w"> </span><span class="m">369116</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">101</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">100</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
<span class="c1"># Here's when we start the Docker container.</span>
<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">185252</span><span class="w"> </span><span class="m">56072</span><span class="w"> </span><span class="m">497736</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">9159</span><span class="w"> </span><span class="m">8016</span><span class="w"> </span><span class="m">2126</span><span class="w"> </span><span class="m">7829</span><span class="w"> </span><span class="m">22</span><span class="w"> </span><span class="m">14</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">21</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">95416</span><span class="w"> </span><span class="m">56576</span><span class="w"> </span><span class="m">525740</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1924</span><span class="w"> </span><span class="m">1036</span><span class="w"> </span><span class="m">396</span><span class="w"> </span><span class="m">1467</span><span class="w"> </span><span class="m">51</span><span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="m">30</span><span class="w"> </span><span class="m">7</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">147764</span><span class="w"> </span><span class="m">56616</span><span class="w"> </span><span class="m">527408</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">336</span><span class="w"> </span><span class="m">630</span><span class="w"> </span><span class="m">117</span><span class="w"> </span><span class="m">307</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">96</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">65604</span><span class="w"> </span><span class="m">56664</span><span class="w"> </span><span class="m">537452</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">82</span><span class="w"> </span><span class="m">81</span><span class="w"> </span><span class="m">494</span><span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="m">94</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">65556</span><span class="w"> </span><span class="m">56676</span><span class="w"> </span><span class="m">537500</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">59</span><span class="w"> </span><span class="m">59</span><span class="w"> </span><span class="m">181</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">99</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">65524</span><span class="w"> </span><span class="m">56684</span><span class="w"> </span><span class="m">537504</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">58</span><span class="w"> </span><span class="m">56</span><span class="w"> </span><span class="m">179</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">99</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">65400</span><span class="w"> </span><span class="m">56692</span><span class="w"> </span><span class="m">537552</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">4928</span><span class="w"> </span><span class="m">138</span><span class="w"> </span><span class="m">231</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">98</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">65400</span><span class="w"> </span><span class="m">56700</span><span class="w"> </span><span class="m">537552</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">3187</span><span class="w"> </span><span class="m">107</span><span class="w"> </span><span class="m">201</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">98</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">0</span>
</code></pre></div>
<p>Interesting. As you can see the <code>id</code> column value decreases. Less idleness, we're starting things. But after a while, the Docker container has started and we're back to things being quiet.</p>
<p>Let's see what <code>free</code> is saying:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>free<span class="w"> </span>-m
<span class="w"> </span>total<span class="w"> </span>used<span class="w"> </span>free<span class="w"> </span>shared<span class="w"> </span>buff/cache<span class="w"> </span>available
Mem:<span class="w"> </span><span class="m">990</span><span class="w"> </span><span class="m">350</span><span class="w"> </span><span class="m">59</span><span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">580</span><span class="w"> </span><span class="m">569</span>
Swap:<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
</code></pre></div>
<p>So, <code>used</code> went up from 127MB to 350MB. The buffer cache also went up. Less available memory.</p>
<p>Let's start the second Docker container. We're looking at vmstat again.</p>
<div class="highlight"><pre><span></span><code>procs<span class="w"> </span>-----------memory----------<span class="w"> </span>---swap--<span class="w"> </span>-----io----<span class="w"> </span>-system--<span class="w"> </span>------cpu-----
<span class="w"> </span>r<span class="w"> </span>b<span class="w"> </span>swpd<span class="w"> </span>free<span class="w"> </span>buff<span class="w"> </span>cache<span class="w"> </span>si<span class="w"> </span>so<span class="w"> </span>bi<span class="w"> </span>bo<span class="w"> </span><span class="k">in</span><span class="w"> </span>cs<span class="w"> </span>us<span class="w"> </span>sy<span class="w"> </span>id<span class="w"> </span>wa<span class="w"> </span>st
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">61304</span><span class="w"> </span><span class="m">56740</span><span class="w"> </span><span class="m">537872</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">56</span><span class="w"> </span><span class="m">83</span><span class="w"> </span><span class="m">227</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">99</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span>
<span class="c1"># Starting second container here.</span>
<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">126516</span><span class="w"> </span><span class="m">59292</span><span class="w"> </span><span class="m">341252</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">7109</span><span class="w"> </span><span class="m">8100</span><span class="w"> </span><span class="m">823</span><span class="w"> </span><span class="m">3255</span><span class="w"> </span><span class="m">22</span><span class="w"> </span><span class="m">15</span><span class="w"> </span><span class="m">56</span><span class="w"> </span><span class="m">6</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">47572</span><span class="w"> </span><span class="m">59904</span><span class="w"> </span><span class="m">337152</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">9186</span><span class="w"> </span><span class="m">7515</span><span class="w"> </span><span class="m">882</span><span class="w"> </span><span class="m">2316</span><span class="w"> </span><span class="m">51</span><span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="m">17</span><span class="w"> </span><span class="m">19</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">153212</span><span class="w"> </span><span class="m">59940</span><span class="w"> </span><span class="m">307720</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1094</span><span class="w"> </span><span class="m">2122</span><span class="w"> </span><span class="m">214</span><span class="w"> </span><span class="m">557</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="m">92</span><span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">92848</span><span class="w"> </span><span class="m">59996</span><span class="w"> </span><span class="m">303368</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">556</span><span class="w"> </span><span class="m">149</span><span class="w"> </span><span class="m">147</span><span class="w"> </span><span class="m">690</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="m">93</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">92848</span><span class="w"> </span><span class="m">60004</span><span class="w"> </span><span class="m">303368</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">108</span><span class="w"> </span><span class="m">91</span><span class="w"> </span><span class="m">311</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">99</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">92724</span><span class="w"> </span><span class="m">60012</span><span class="w"> </span><span class="m">303376</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">102</span><span class="w"> </span><span class="m">94</span><span class="w"> </span><span class="m">326</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">98</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">88296</span><span class="w"> </span><span class="m">60028</span><span class="w"> </span><span class="m">305048</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">320</span><span class="w"> </span><span class="m">102</span><span class="w"> </span><span class="m">147</span><span class="w"> </span><span class="m">424</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">96</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">77228</span><span class="w"> </span><span class="m">60364</span><span class="w"> </span><span class="m">314520</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1876</span><span class="w"> </span><span class="m">144</span><span class="w"> </span><span class="m">306</span><span class="w"> </span><span class="m">868</span><span class="w"> </span><span class="m">28</span><span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="m">64</span><span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">86800</span><span class="w"> </span><span class="m">60372</span><span class="w"> </span><span class="m">304968</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">26</span><span class="w"> </span><span class="m">387</span><span class="w"> </span><span class="m">111</span><span class="w"> </span><span class="m">372</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">97</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">86032</span><span class="w"> </span><span class="m">60780</span><span class="w"> </span><span class="m">305120</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">110</span><span class="w"> </span><span class="m">119</span><span class="w"> </span><span class="m">150</span><span class="w"> </span><span class="m">399</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">98</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">0</span>
</code></pre></div>
<p>Similar situation. A spike in io, a spike in non-idle CPU percentage, followed by quiet. Let's look at <code>free</code>:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>free<span class="w"> </span>-m
<span class="w"> </span>total<span class="w"> </span>used<span class="w"> </span>free<span class="w"> </span>shared<span class="w"> </span>buff/cache<span class="w"> </span>available
Mem:<span class="w"> </span><span class="m">990</span><span class="w"> </span><span class="m">549</span><span class="w"> </span><span class="m">84</span><span class="w"> </span><span class="m">17</span><span class="w"> </span><span class="m">356</span><span class="w"> </span><span class="m">371</span>
Swap:<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span>
</code></pre></div>
<p>More used memory. But notice how this time the <code>buff/cache</code> section went down. Why? Well, because our running processes are using more memory and this memory has to come from somewhere. The kernel is freeing memory from the buffer cache and giving it to processes. Again, reasonable behavior.</p>
<p>Let's start the third and fourth Docker containers and see how <code>vmstat</code> looks after that.</p>
<div class="highlight"><pre><span></span><code>procs<span class="w"> </span>-----------memory----------<span class="w"> </span>---swap--<span class="w"> </span>-----io----<span class="w"> </span>-system--<span class="w"> </span>------cpu-----
<span class="w"> </span>r<span class="w"> </span>b<span class="w"> </span>swpd<span class="w"> </span>free<span class="w"> </span>buff<span class="w"> </span>cache<span class="w"> </span>si<span class="w"> </span>so<span class="w"> </span>bi<span class="w"> </span>bo<span class="w"> </span><span class="k">in</span><span class="w"> </span>cs<span class="w"> </span>us<span class="w"> </span>sy<span class="w"> </span>id<span class="w"> </span>wa<span class="w"> </span>st
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">46052</span><span class="w"> </span><span class="m">11188</span><span class="w"> </span><span class="m">160564</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">151</span><span class="w"> </span><span class="m">153</span><span class="w"> </span><span class="m">532</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">97</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">39828</span><span class="w"> </span><span class="m">11336</span><span class="w"> </span><span class="m">166664</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1246</span><span class="w"> </span><span class="m">141</span><span class="w"> </span><span class="m">235</span><span class="w"> </span><span class="m">634</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">95</span><span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">39704</span><span class="w"> </span><span class="m">11344</span><span class="w"> </span><span class="m">166664</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">149</span><span class="w"> </span><span class="m">158</span><span class="w"> </span><span class="m">542</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">97</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">0</span>
<span class="c1"># After we start the fourth container.</span>
<span class="w"> </span><span class="m">6</span><span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">18840</span><span class="w"> </span><span class="m">11668</span><span class="w"> </span><span class="m">179484</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">18678</span><span class="w"> </span><span class="m">7726</span><span class="w"> </span><span class="m">2711</span><span class="w"> </span><span class="m">7721</span><span class="w"> </span><span class="m">5</span><span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="m">46</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">8964</span><span class="w"> </span><span class="m">2796</span><span class="w"> </span><span class="m">103108</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">38137</span><span class="w"> </span><span class="m">2508</span><span class="w"> </span><span class="m">4198</span><span class="w"> </span><span class="m">4872</span><span class="w"> </span><span class="m">37</span><span class="w"> </span><span class="m">21</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">11</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">9576</span><span class="w"> </span><span class="m">588</span><span class="w"> </span><span class="m">101548</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">68988</span><span class="w"> </span><span class="m">485</span><span class="w"> </span><span class="m">4097</span><span class="w"> </span><span class="m">6006</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">88</span><span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">9352</span><span class="w"> </span><span class="m">152</span><span class="w"> </span><span class="m">100752</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">67797</span><span class="w"> </span><span class="m">250</span><span class="w"> </span><span class="m">2463</span><span class="w"> </span><span class="m">4497</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">90</span><span class="w"> </span><span class="m">1</span>
....
<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">18</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">9508</span><span class="w"> </span><span class="m">248</span><span class="w"> </span><span class="m">100560</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">63568</span><span class="w"> </span><span class="m">138</span><span class="w"> </span><span class="m">1700</span><span class="w"> </span><span class="m">3935</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">91</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">14</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">8656</span><span class="w"> </span><span class="m">400</span><span class="w"> </span><span class="m">100436</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">63154</span><span class="w"> </span><span class="m">68</span><span class="w"> </span><span class="m">1580</span><span class="w"> </span><span class="m">3766</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">91</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">19</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">8852</span><span class="w"> </span><span class="m">288</span><span class="w"> </span><span class="m">100552</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">63711</span><span class="w"> </span><span class="m">128</span><span class="w"> </span><span class="m">1606</span><span class="w"> </span><span class="m">3705</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">92</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">47</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">10232</span><span class="w"> </span><span class="m">372</span><span class="w"> </span><span class="m">98616</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">63410</span><span class="w"> </span><span class="m">94</span><span class="w"> </span><span class="m">1720</span><span class="w"> </span><span class="m">4316</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">7</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">91</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">50</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">9648</span><span class="w"> </span><span class="m">420</span><span class="w"> </span><span class="m">99604</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">63126</span><span class="w"> </span><span class="m">77</span><span class="w"> </span><span class="m">1696</span><span class="w"> </span><span class="m">4425</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">91</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">40</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">10028</span><span class="w"> </span><span class="m">272</span><span class="w"> </span><span class="m">99432</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">63692</span><span class="w"> </span><span class="m">74</span><span class="w"> </span><span class="m">1681</span><span class="w"> </span><span class="m">4506</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">7</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">91</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">75</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">9944</span><span class="w"> </span><span class="m">160</span><span class="w"> </span><span class="m">98220</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">63342</span><span class="w"> </span><span class="m">59</span><span class="w"> </span><span class="m">1724</span><span class="w"> </span><span class="m">5150</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">89</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="m">34</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">8952</span><span class="w"> </span><span class="m">296</span><span class="w"> </span><span class="m">100400</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">63575</span><span class="w"> </span><span class="m">110</span><span class="w"> </span><span class="m">1958</span><span class="w"> </span><span class="m">4663</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">91</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">5</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">9004</span><span class="w"> </span><span class="m">1716</span><span class="w"> </span><span class="m">117640</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">62704</span><span class="w"> </span><span class="m">123</span><span class="w"> </span><span class="m">2674</span><span class="w"> </span><span class="m">5329</span><span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="m">9</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">88</span><span class="w"> </span><span class="m">1</span>
</code></pre></div>
<p>This is where things start to get bad! The box becomes very unresponsive. Typing a simple command like <code>ls</code> takes seconds to execute. Let's take a peek at <code>free</code> and we'll continue to our analysis after that:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>free<span class="w"> </span>-m
<span class="w"> </span>total<span class="w"> </span>used<span class="w"> </span>free<span class="w"> </span>shared<span class="w"> </span>buff/cache<span class="w"> </span>available
Mem:<span class="w"> </span><span class="m">990</span><span class="w"> </span><span class="m">851</span><span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">24</span><span class="w"> </span><span class="m">119</span><span class="w"> </span><span class="m">54</span>
Swap:<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
</code></pre></div>
<p>As you can see, we barely have any free memory and the buffer cache has shrunk even more. But notice what's going on in <code>vmstat</code>. The <code>bi</code> (blocks received from a block device) is stuck at a constant high value. This means we're doing a lot of disk reads. Why is that, are our processes doing a lot of disk operations? No. Our many processes are executables that are located on disk. When the kernel executes them, it has to read instructions. If we have enough buffer cache, we can store these instruction in memory and not have to read them again. However, our buffer cache is small now. So when processes run, the kernel needs to pull their instructions from disk. It probably stores them in the buffer cache, but when the next process is running, it tries to store in the cache again and evicts what's stored from the old process. </p>
<p>Are we done? Not yet. When this whole sad mess happens, the kernel will run something called <a href="https://linux-mm.org/OOM_Killer">OOM killer</a>. How do we know this is what's going on? We can use <code>dmesg</code> and view the system messages:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>dmesg<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">"Out of memory"</span>
<span class="o">[</span><span class="w"> </span><span class="m">3635</span>.537538<span class="o">]</span><span class="w"> </span>Out<span class="w"> </span>of<span class="w"> </span>memory:<span class="w"> </span>Kill<span class="w"> </span>process<span class="w"> </span><span class="m">21580</span><span class="w"> </span><span class="o">(</span>jupyter-noteboo<span class="o">)</span><span class="w"> </span>score<span class="w"> </span><span class="m">37</span><span class="w"> </span>or<span class="w"> </span>sacrifice<span class="w"> </span>child
<span class="o">[</span><span class="w"> </span><span class="m">3636</span>.822607<span class="o">]</span><span class="w"> </span>Out<span class="w"> </span>of<span class="w"> </span>memory:<span class="w"> </span>Kill<span class="w"> </span>process<span class="w"> </span><span class="m">22714</span><span class="w"> </span><span class="o">(</span>jupyter-noteboo<span class="o">)</span><span class="w"> </span>score<span class="w"> </span><span class="m">37</span><span class="w"> </span>or<span class="w"> </span>sacrifice<span class="w"> </span>child
<span class="o">[</span><span class="w"> </span><span class="m">3643</span>.006328<span class="o">]</span><span class="w"> </span>Out<span class="w"> </span>of<span class="w"> </span>memory:<span class="w"> </span>Kill<span class="w"> </span>process<span class="w"> </span><span class="m">24976</span><span class="w"> </span><span class="o">(</span>jupyter-noteboo<span class="o">)</span><span class="w"> </span>score<span class="w"> </span><span class="m">37</span><span class="w"> </span>or<span class="w"> </span>sacrifice<span class="w"> </span>child
<span class="o">[</span><span class="w"> </span><span class="m">3654</span>.916468<span class="o">]</span><span class="w"> </span>Out<span class="w"> </span>of<span class="w"> </span>memory:<span class="w"> </span>Kill<span class="w"> </span>process<span class="w"> </span><span class="m">26118</span><span class="w"> </span><span class="o">(</span>jupyter-noteboo<span class="o">)</span><span class="w"> </span>score<span class="w"> </span><span class="m">37</span><span class="w"> </span>or<span class="w"> </span>sacrifice<span class="w"> </span>child
<span class="o">[</span><span class="w"> </span><span class="m">3658</span>.712286<span class="o">]</span><span class="w"> </span>Out<span class="w"> </span>of<span class="w"> </span>memory:<span class="w"> </span>Kill<span class="w"> </span>process<span class="w"> </span><span class="m">28364</span><span class="w"> </span><span class="o">(</span>jupyter-noteboo<span class="o">)</span><span class="w"> </span>score<span class="w"> </span><span class="m">35</span><span class="w"> </span>or<span class="w"> </span>sacrifice<span class="w"> </span>child
<span class="o">[</span><span class="w"> </span><span class="m">3666</span>.654763<span class="o">]</span><span class="w"> </span>Out<span class="w"> </span>of<span class="w"> </span>memory:<span class="w"> </span>Kill<span class="w"> </span>process<span class="w"> </span><span class="m">30603</span><span class="w"> </span><span class="o">(</span>jupyter-noteboo<span class="o">)</span><span class="w"> </span>score<span class="w"> </span><span class="m">36</span><span class="w"> </span>or<span class="w"> </span>sacrifice<span class="w"> </span>child
<span class="o">[</span><span class="w"> </span><span class="m">3685</span>.390829<span class="o">]</span><span class="w"> </span>Out<span class="w"> </span>of<span class="w"> </span>memory:<span class="w"> </span>Kill<span class="w"> </span>process<span class="w"> </span><span class="m">30620</span><span class="w"> </span><span class="o">(</span>jupyter-noteboo<span class="o">)</span><span class="w"> </span>score<span class="w"> </span><span class="m">36</span><span class="w"> </span>or<span class="w"> </span>sacrifice<span class="w"> </span>child
</code></pre></div>
<p>The strange thing here is that it keeps killing these <a href="http://jupyter.org/">Jupyter</a> processes for a long time. My guess here is that something restarts them after they're killed.</p>
<p>The best thing to do here is to simply kill all the running Docker containers so that the box is usable again:</p>
<div class="highlight"><pre><span></span><code>docker<span class="w"> </span><span class="nb">kill</span><span class="w"> </span><span class="k">$(</span>docker<span class="w"> </span>ps<span class="w"> </span>-q<span class="k">)</span>
</code></pre></div>
<p>Things would look different if there was a swap partition on the disk. I might try such a setup as well. If you think about it, the swap would allow some of the memory used by processes to be transferred to disk. That's a great win, because some parts of memory might be very rarely if at all accessed. However, without a swap, there's no such option.</p>The Cloud Infrastructure Landscape2017-01-15T12:10:00-08:002017-01-15T12:10:00-08:00Petko Minkovtag:pminkov.github.io,2017-01-15:/blog/the-cloud-infrastructure-landscape.html<p>I'm taking a <a href="https://www.edx.org/course/introduction-cloud-infrastructure-linuxfoundationx-lfs151-x">Cloud Infrastructure</a> class on EdX and I have found it to be a really nice overview of the cloud space. It's missing anything related to Hadoop and data analytics, but it describes probably all the tools related to running applications in the cloud. Of course, since the …</p><p>I'm taking a <a href="https://www.edx.org/course/introduction-cloud-infrastructure-linuxfoundationx-lfs151-x">Cloud Infrastructure</a> class on EdX and I have found it to be a really nice overview of the cloud space. It's missing anything related to Hadoop and data analytics, but it describes probably all the tools related to running applications in the cloud. Of course, since the class describes so many tools, it can't do so in depth, but you can pick whatever is interesting to you and learn more about it.</p>
<p>Here's a list of all the tools described in it, which list I might use as a future reference for myself. I'd say that knowing a bit about such a big amount of products definitely helps when you're trying to come up with a cloud infrastructure for a project.</p>
<p>So, big list, based on the course contents:</p>
<h3 id="virtualization">Virtualization</h3>
<p><a href="http://www.linux-kvm.org/page/Main_Page">KVM</a><br>
<a href="https://www.virtualbox.org/wiki/VirtualBox">VirtualBox</a><br>
<a href="https://www.vagrantup.com/">Vagrant</a><br></p>
<p>I use VirtualBox to run Linux on Mac OS X and highly recommend it. Mac OS X is not good if you're trying to learn Linux, too many differences.</p>
<h3 id="infrastructure-as-a-service-iaas">Infrastructure as a Service (IaaS)<br></h3>
<p><a href="https://aws.amazon.com/ec2/">Amazon EC2</a><br>
<a href="https://azure.microsoft.com/en-us/services/virtual-machines/?b=16.51a">Azure Virtual Machines</a><br>
<a href="https://www.digitalocean.com/">Digital Ocean</a><br>
<a href="https://cloud.google.com/compute/">Google Compute Engine</a><br>
<a href="https://www.openstack.org/">OpenStack</a><br></p>
<p>I've only tried EC2 out of these, but it's pretty simple and I like the fact that you can do so many things through the UI, it makes it easier to start with it.</p>
<h3 id="platform-as-a-sevice-paas">Platform as a Sevice (PaaS)<br></h3>
<p><a href="https://www.cloudfoundry.org/learn/features/">Cloud Foundry</a><br>
<a href="https://www.openshift.com/">OpenShift</a><br>
<a href="https://www.heroku.com/">Heroku</a><br>
<a href="https://deis.com/">Deis</a><br>
<a href="https://aws.amazon.com/elasticbeanstalk/">AWS Elastic Beanstalk</a> (my addition)<br></p>
<p>I've used Elastic Beanstalk and it's a good way to abstract configuration of load balancers, Apache instances and so on.</p>
<h3 id="containers">Containers</h3>
<p><a href="https://linuxcontainers.org/">LXC (Linux Containers)</a><br>
<a href="https://www.docker.com/">Docker</a><br></p>
<p>Docker is the leader here. I'm surprised that companies like Google or VMWare haven't come up with their own solution. Maybe soon.</p>
<h3 id="micro-oses-for-containers">Micro OSes for Containers</h3>
<p><a href="https://www.projectatomic.io/">Atomic Host</a><br>
<a href="https://www.coreos.com/">CoreOS</a><br>
<a href="https://vmware.github.io/photon/">VMWare Photon</a><br>
<a href="http://rancher.com/rancher-os/">RancherOS</a><br></p>
<p>Pretty neat concept of building a minimal OS that's targeted towards running containers.</p>
<h3 id="container-orchestration">Container Orchestration</h3>
<p><a href="https://www.docker.com/products/docker-swarm">Docker Swarm</a><br>
<a href="https://kubernetes.io/">Kubernetes</a><br></p>
<h3 id="unikernels">Unikernels</h3>
<p><a href="https://github.com/emc-advanced-dev/unik">Unik</a><br></p>
<h3 id="container-as-a-service-caas">Container as a Service (CaaS)<br></h3>
<p><a href="https://www.docker.com/products/docker-datacenter">Docker Datacenter</a><br>
<a href="https://wiki.openstack.org/wiki/Magnum">Project Magnum on OpenStack</a><br></p>
<h3 id="software-defined-storage-and-storage-management-for-containers">Software Defined Storage and Storage Management for Containers</h3>
<p><a href="http://ceph.com/">Ceph</a><br>
<a href="https://www.gluster.org/">Gluster</a><br>
<a href="https://storpool.com/">StorPool</a> (My addition)<br> </p>
<h3 id="continuous-integration-continuous-delivery">Continuous Integration / Continuous Delivery</h3>
<p><a href="https://jenkins.io/">Jenkins</a><br>
<a href="https://drone.io/">Drone</a><br>
<a href="https://travis-ci.com/getting_started">Travis CI</a><br>
<a href="https://app.shippable.com/">Shippable</a><br></p>
<p>These are very easy to try out if you have projects on GitHub. Here's <a href="https://github.com/pminkov/webserver">one</a> on which I slapped the Drone badge.</p>
<h3 id="tools-for-configuration-management">Tools for Configuration Management</h3>
<p><a href="https://www.ansible.com/">Ansible</a><br>
<a href="https://puppet.com/">Puppet</a><br>
<a href="https://docs.saltstack.com/en/latest/">SaltStack</a><br>
<a href="https://www.chef.io/">Chef</a><br></p>
<p>Useful tools for running commands on multiple servers.</p>
<h3 id="build-and-release-tools">Build and Release Tools</h3>
<p><a href="https://www.terraform.io/docs/providers/">Terraform</a><br>
<a href="https://bosh.io">BOSH</a><br></p>
<h3 id="key-value-pair-stores">Key-Value Pair Stores</h3>
<p><a href="https://coreos.com/etcd/">etcd</a><br>
<a href="https://www.consul.io/">Consul</a><br></p>
<h3 id="building-images">Building Images</h3>
<p><a href="https://www.packer.io/">Packer</a><br></p>
<p>Pretty neat. Just describe your instance and build a VM / Amazon image / Docker container. I can see that being pretty useful when you're building AMIs for AWS. Better than building it manually and not being able to reproduce it on another platform.</p>
<h3 id="debugging-logging-monitoring">Debugging, Logging, Monitoring</h3>
<p><a href="https://www.sysdig.com/">Sysdig</a><br>
<a href="https://github.com/google/cadvisor">cAdvisor</a><br>
<a href="http://www.fluentd.org/">Fluentd</a><br>
<a href="https://www.datadoghq.com/">Datadog</a><br></p>How to install collectd on Ubuntu.2017-01-05T14:46:00-08:002017-01-05T14:46:00-08:00Petko Minkovtag:pminkov.github.io,2017-01-05:/blog/how-to-install-collectd-on-ubuntu.html<p>Some time ago I found out about <a href="https://collectd.org/">collectd</a> and I was curious to see what it does. collectd collects statistics about the machine its running on - cpu, disk, memory, processes, battery, etc. </p>
<p>Here's how to install it on Ubuntu and visualize the data it has collected.</p>
<p><br></p>
<h3 id="step-1-install-the-collectd-package">Step 1: Install the …</h3><p>Some time ago I found out about <a href="https://collectd.org/">collectd</a> and I was curious to see what it does. collectd collects statistics about the machine its running on - cpu, disk, memory, processes, battery, etc. </p>
<p>Here's how to install it on Ubuntu and visualize the data it has collected.</p>
<p><br></p>
<h3 id="step-1-install-the-collectd-package">Step 1: Install the collectd package.</h3>
<p>Easy, just install the package:<br></p>
<p><code>sudo apt-get install collectd</code></p>
<p><br></p>
<h3 id="step-2-make-sure-collectd-and-apache-are-running">Step 2: Make sure collectd and apache are running.</h3>
<p>If you have installed apache, you should have both collectd and apache running</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>service<span class="w"> </span>--status-all<span class="w"> </span><span class="p">|</span><span class="w"> </span>egrep<span class="w"> </span><span class="s2">"collectd|apache2"</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span>+<span class="w"> </span><span class="o">]</span><span class="w"> </span>apache2
<span class="w"> </span><span class="o">[</span><span class="w"> </span>+<span class="w"> </span><span class="o">]</span><span class="w"> </span>collectd
</code></pre></div>
<p>If collectd is not running, run <code>sudo service collectd start</code>. For me at least, it was running after installation.</p>
<p><br></p>
<h3 id="step-3-install-collectds-web-app-for-generating-graphs">Step 3: Install collectd's web app for generating graphs.</h3>
<p>Ok, now we have collectd running. collectd is mostly about collecting data and it allows other frontends to display it. However, it comes with a simple set of cgi scripts that can be used to see some graphs.</p>
<p>In the <code>/usr/share/doc/collectd/examples/</code> directory, you'll find a directory named <code>collection3</code>. Copy the entire directory to <code>/var/www/html</code>:</p>
<p><code>$ sudo cp -r ./collection3 /var/www/html</code>.</p>
<p><br></p>
<h3 id="step-4-enable-apache-to-run-cgi-scripts">Step 4: Enable apache to run CGI scripts.</h3>
<p>Great, you can now access the cgi scripts by going to this url: <code>http://localhost/collection3/bin/index.cgi</code>. However, you'll be served a text file, since apache doesn't know to run these cgi scripts. There's is a <a href="http://httpd.apache.org/docs/2.2/howto/cgi.html">simple manual</a> explaining cgi scripts in Apache.</p>
<p>You'll have to do two things.</p>
<p>First, you need to install the cgi module. So, go to <code>/etc/apache2/mods-enabled</code> and run this: <code>$ sudo ln -s ../mods-available/cgi.load</code>. You have now enabled the <code>cgi</code> module.</p>
<p>Next you'll have to change <code>apache2.conf</code>, located in <code>/etc/apache2</code> (Ubuntu doesn't use <code>httpd.conf</code>).</p>
<p>Add these lines to it:</p>
<div class="highlight"><pre><span></span><code><Directory /var/www/>
Options +ExecCGI
AddHandler cgi-script .cgi
</Directory>
</code></pre></div>
<p>And - you're done! If you go to <code>http://localhost/cgi-bin/collection3/bin/index.cgi</code>, you should see some graphs.</p>Thoughts on C programming.2017-01-01T17:58:00-08:002017-01-01T17:58:00-08:00Petko Minkovtag:pminkov.github.io,2017-01-01:/blog/thoughts-on-c-programming.html<p>I recently wrote a <a href="https://github.com/pminkov/webserver">webserver</a> in C and wanted to share my thoughts on writing code in C. This was practically my first somewhat substantial C project. I've had two jobs in which I was writing C++, but I never programmed in C.</p>
<p>So, in no particular order, here are …</p><p>I recently wrote a <a href="https://github.com/pminkov/webserver">webserver</a> in C and wanted to share my thoughts on writing code in C. This was practically my first somewhat substantial C project. I've had two jobs in which I was writing C++, but I never programmed in C.</p>
<p>So, in no particular order, here are my thoughts on C.
<br>
<br>
<br></p>
<h3 id="the-language-is-simple">The language is simple.</h3>
<p>C is a pretty simple language. Unlike complicated languages like Scala, C has a very limited set of language features. There's a good side to that - you can start faster with C and I appreciate the simplicity coming from working with a more limited language. On the other hand, you have to write more code to do certain things.
<br>
<br>
<br></p>
<h3 id="lack-of-object-oriented-programming-makes-writing-code-more-difficult">Lack of object oriented programming makes writing code more difficult.</h3>
<p>It's difficult to organize code when you don't have classes. It seems like your option is to just put pieces of code in separate files if you want to do that. But then, function names will still have to be different. So you end up with a lot of functions containing the name of the type they're supposed to operate on, like maybe <code>add_to_tree</code> and then <code>remove_from_tree</code> instead of simple <code>add</code> and <code>remove</code> methods on a <code>Tree</code> class.
<br>
<br>
<br></p>
<h3 id="not-knowing-the-size-of-arrays-and-strings-in-advance-is-difficult">Not knowing the size of arrays and strings in advance is difficult.</h3>
<p>Unlike C++, C doesn't have its STL library, so if you're allocating arrays or strings, you often have to know their size in advance. Imagine running a process with <code>popen</code> and having to store its output to a string. You can't know the size of the output in advance, so you have to allocate a buffer of a certain size, expecting it to be big enough in all cases. That's of course impossible, so you'll need to code your own variable size string.</p>
<p>C library functions are also not immune to this problem. For example, the <code>getcwd</code> is a function that gets the current working directory. It receives a buffer of a certain size and writes the directory name to it. If the buffer is smaller than the directory name, the function will return NULL.</p>
<p>Luckily, in a lot of cases, you'd know the size of the data in advance, but writing code that has to do with strings and arrays is still more difficult than just using a STL <code>vector</code> or a <code>stack</code>.
<br>
<br>
<br></p>
<h3 id="you-have-to-be-meticulous-in-checking-return-codes-of-standard-library-functions">You have to be meticulous in checking return codes of standard library functions.</h3>
<p>Most of the functions in the C standard library return something that indicates the success or failure of the function. Unlike languages that have exceptions, if you don't check the return values, it's entirely possible that your code will continue to execute and ignore some errors. Since often times you'll find yourself thinking "I doubt this can fail", writing good C code requires an extra level of discipline.</p>How to fix order-violation bugs with condition variables.2016-11-29T11:22:00-08:002016-11-29T11:22:00-08:00Petko Minkovtag:pminkov.github.io,2016-11-29:/blog/how-to-fix-order-violation-bugs-with-condition-variables.html<p>Let's look at the following code and try to find the bug in it.</p>
<div class="highlight"><pre><span></span><code><span class="nt">Thread</span><span class="w"> </span><span class="nt">1</span><span class="o">:</span>
<span class="nt">void</span><span class="w"> </span><span class="nt">init</span><span class="o">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">...</span>
<span class="w"> </span><span class="err">mThread</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="err">PR_CreateThread(mMain,</span><span class="w"> </span><span class="err">...)</span>
<span class="w"> </span><span class="err">...</span>
<span class="p">}</span>
<span class="nt">Thread</span><span class="w"> </span><span class="nt">2</span><span class="o">:</span>
<span class="nt">void</span><span class="w"> </span><span class="nt">mMain</span><span class="o">(...)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">...</span>
<span class="w"> </span><span class="err">mState</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="err">mThread->State</span><span class="p">;</span>
<span class="w"> </span><span class="err">...</span>
<span class="p">}</span>
</code></pre></div>
<p>The problem is that if thread two executes before thread one, it can access <code>mThread</code> which is not initialized. We need to …</p><p>Let's look at the following code and try to find the bug in it.</p>
<div class="highlight"><pre><span></span><code><span class="nt">Thread</span><span class="w"> </span><span class="nt">1</span><span class="o">:</span>
<span class="nt">void</span><span class="w"> </span><span class="nt">init</span><span class="o">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">...</span>
<span class="w"> </span><span class="err">mThread</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="err">PR_CreateThread(mMain,</span><span class="w"> </span><span class="err">...)</span>
<span class="w"> </span><span class="err">...</span>
<span class="p">}</span>
<span class="nt">Thread</span><span class="w"> </span><span class="nt">2</span><span class="o">:</span>
<span class="nt">void</span><span class="w"> </span><span class="nt">mMain</span><span class="o">(...)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">...</span>
<span class="w"> </span><span class="err">mState</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="err">mThread->State</span><span class="p">;</span>
<span class="w"> </span><span class="err">...</span>
<span class="p">}</span>
</code></pre></div>
<p>The problem is that if thread two executes before thread one, it can access <code>mThread</code> which is not initialized. We need to make thread two wait for thread one. This is done with condition variables. But let's see how we're going to implement it.</p>
<h3 id="solution-1-incorrect-simple-wait-and-signal">Solution 1 (Incorrect) - Simple wait and signal</h3>
<p>Let's say that we're not giving this a lot of thought and just put a wait and signal in the code. We have this code:</p>
<div class="highlight"><pre><span></span><code><span class="n">pthread_mutex_t</span><span class="w"> </span><span class="n">mtLock</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">PTHREAD_MUTEX_INITIALIZER</span><span class="p">;</span>
<span class="n">pthread_cond_t</span><span class="w"> </span><span class="n">mtCond</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">PTHREAD_COND_INITIALIZER</span><span class="p">;</span>
<span class="n">Thread</span><span class="w"> </span><span class="mi">1</span><span class="p">:</span>
<span class="nb nb-Type">void</span><span class="w"> </span><span class="n">init</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="o">...</span>
<span class="w"> </span><span class="n">mThread</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">PR_CreateThread</span><span class="p">(</span><span class="n">mMain</span><span class="p">,</span><span class="w"> </span><span class="o">...</span><span class="p">)</span>
<span class="w"> </span><span class="n">pthread_mutex_lock</span><span class="p">(</span><span class="o">&</span><span class="n">mtLock</span><span class="p">);</span>
<span class="w"> </span><span class="n">pthread_cond_signal</span><span class="p">(</span><span class="o">&</span><span class="n">mtCond</span><span class="p">);</span>
<span class="w"> </span><span class="n">pthread_mutex_unlock</span><span class="p">(</span><span class="o">&</span><span class="n">mtLock</span><span class="p">);</span>
<span class="w"> </span><span class="o">...</span>
<span class="p">}</span>
<span class="n">Thread</span><span class="w"> </span><span class="mi">2</span><span class="p">:</span>
<span class="nb nb-Type">void</span><span class="w"> </span><span class="n">mMain</span><span class="p">(</span><span class="o">...</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="o">...</span>
<span class="w"> </span><span class="n">pthread_mutex_lock</span><span class="p">(</span><span class="o">&</span><span class="n">mtLock</span><span class="p">);</span>
<span class="w"> </span><span class="n">pthread_cond_wait</span><span class="p">(</span><span class="o">&</span><span class="n">mtCond</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">mtLock</span><span class="p">);</span>
<span class="w"> </span><span class="n">pthread_mutex_unlock</span><span class="p">(</span><span class="o">&</span><span class="n">mtLock</span><span class="p">);</span>
<span class="w"> </span><span class="n">mState</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mThread</span><span class="o">-></span><span class="n">State</span><span class="p">;</span>
<span class="w"> </span><span class="o">...</span>
<span class="p">}</span>
</code></pre></div>
<p>The problem here is that thread one can execute before thread two. In this case, thread one will signal on the condition variable, but nobody is waiting on it. Once thread two executes, it will start waiting and nobody is going to signal on the condition variable. Thus, thread two will stay in block state.</p>
<h3 id="solution-2-correct-using-a-synchronization-variable">Solution 2 (Correct) - Using a synchronization variable.</h3>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="n">pthread_mutex_t</span><span class="w"> </span><span class="n">mtLock</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">PTHREAD_MUTEX_INITIALIZER</span><span class="p">;</span>
<span class="w"> </span><span class="n">pthread_cond_t</span><span class="w"> </span><span class="n">mtCond</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">PTHREAD_COND_INITIALIZER</span><span class="p">;</span>
<span class="w"> </span><span class="nb nb-Type">int</span><span class="w"> </span><span class="n">mtInit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="n">Thread</span><span class="w"> </span><span class="mi">1</span><span class="p">:</span>
<span class="w"> </span><span class="nb nb-Type">void</span><span class="w"> </span><span class="n">init</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="o">...</span>
<span class="mi">1</span><span class="w"> </span><span class="n">mThread</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">PR_CreateThread</span><span class="p">(</span><span class="n">mMain</span><span class="p">,</span><span class="w"> </span><span class="o">...</span><span class="p">)</span>
<span class="mi">2</span><span class="w"> </span><span class="n">pthread_mutex_lock</span><span class="p">(</span><span class="o">&</span><span class="n">mtLock</span><span class="p">);</span>
<span class="mi">3</span><span class="w"> </span><span class="n">mtInit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span>
<span class="mi">4</span><span class="w"> </span><span class="n">pthread_cond_signal</span><span class="p">(</span><span class="o">&</span><span class="n">mtCond</span><span class="p">);</span>
<span class="mi">5</span><span class="w"> </span><span class="n">pthread_mutex_unlock</span><span class="p">(</span><span class="o">&</span><span class="n">mtLock</span><span class="p">);</span>
<span class="w"> </span><span class="o">...</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">Thread</span><span class="w"> </span><span class="mi">2</span><span class="p">:</span>
<span class="w"> </span><span class="nb nb-Type">void</span><span class="w"> </span><span class="n">mMain</span><span class="p">(</span><span class="o">...</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="o">...</span>
<span class="mi">6</span><span class="w"> </span><span class="n">pthread_mutex_lock</span><span class="p">(</span><span class="o">&</span><span class="n">mtLock</span><span class="p">);</span>
<span class="mi">7</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">mtInit</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span>
<span class="mi">8</span><span class="w"> </span><span class="n">pthread_cond_wait</span><span class="p">(</span><span class="o">&</span><span class="n">mtCond</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">mtLock</span><span class="p">);</span>
<span class="mi">9</span><span class="w"> </span><span class="n">pthread_mutex_unlock</span><span class="p">(</span><span class="o">&</span><span class="n">mtLock</span><span class="p">);</span>
<span class="mi">10</span><span class="w"> </span><span class="n">mState</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mThread</span><span class="o">-></span><span class="n">State</span><span class="p">;</span>
<span class="w"> </span><span class="o">...</span>
<span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>Here we have used the <code>mtInit</code> variable to prevent the bug in Solution 1 from hapenning. Let's try to reproduce the conditions that led to a bug in out first solution. Let's say thread one executes before thread two. When thread two runs, it will see that <code>mtInit</code> is 1 and it won't wait. That basically solves our problem. Let's trace things more carfully with two possible scenarios. Of course, there are other possibilities too, but with these two I have enough confidence in the correctness of the solution.</p>
<div class="highlight"><pre><span></span><code><span class="n">Scenario</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Thread</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">before</span><span class="w"> </span><span class="n">Thread</span><span class="w"> </span><span class="mf">2.</span>
<span class="n">Line</span><span class="w"> </span><span class="n">T1</span><span class="w"> </span><span class="n">T2</span><span class="w"> </span><span class="n">COMMENT</span>
<span class="w"> </span><span class="mi">1</span>
<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Locked</span><span class="w"> </span><span class="n">mutex</span>
<span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="n">Blocked</span><span class="p">,</span><span class="w"> </span><span class="n">because</span><span class="w"> </span><span class="n">mutex</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">locked</span><span class="o">.</span>
<span class="w"> </span><span class="mi">3</span>
<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">Signals</span><span class="w"> </span><span class="n">on</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">condition</span><span class="w"> </span><span class="n">variable</span><span class="o">.</span>
<span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="n">Unblocks</span><span class="w"> </span><span class="n">mutex</span><span class="o">.</span>
<span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="n">Continues</span><span class="p">,</span><span class="w"> </span><span class="n">mutex</span><span class="w"> </span><span class="n">was</span><span class="w"> </span><span class="n">unblocked</span><span class="o">.</span>
<span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">jump</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mf">9.</span>
<span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">Unlock</span><span class="w"> </span><span class="n">mutex</span><span class="p">,</span><span class="w"> </span><span class="n">done</span>
</code></pre></div>
<p>So in this scenario, thread one obtains the lock first and the solution is correct.</p>
<div class="highlight"><pre><span></span><code><span class="n">Scenario</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Thread</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">before</span><span class="w"> </span><span class="n">Thread</span><span class="w"> </span><span class="mf">1.</span>
<span class="n">Line</span><span class="w"> </span><span class="n">T1</span><span class="w"> </span><span class="n">T2</span><span class="w"> </span><span class="n">COMMENT</span>
<span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="n">Locked</span><span class="w"> </span><span class="n">mutex</span><span class="o">.</span>
<span class="w"> </span><span class="mi">7</span>
<span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="n">Unlocks</span><span class="w"> </span><span class="n">mutex</span><span class="p">,</span><span class="w"> </span><span class="n">waiting</span><span class="w"> </span><span class="n">on</span><span class="w"> </span><span class="n">condition</span><span class="w"> </span><span class="n">variable</span><span class="o">.</span>
<span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Locks</span><span class="w"> </span><span class="n">mutex</span><span class="o">.</span>
<span class="w"> </span><span class="mi">3</span><span class="w"> </span>
<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">Signals</span><span class="p">,</span><span class="w"> </span><span class="n">but</span><span class="w"> </span><span class="n">mutex</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">still</span><span class="w"> </span><span class="n">locked</span><span class="p">,</span><span class="w"> </span><span class="n">so</span><span class="w"> </span><span class="n">thread</span><span class="w"> </span><span class="n">two</span><span class="w"> </span><span class="n">doesn</span><span class="s1">'t do anything.</span>
<span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="n">Unlocks</span><span class="w"> </span><span class="n">mutex</span><span class="o">.</span><span class="w"> </span><span class="n">Now</span><span class="w"> </span><span class="n">thread</span><span class="w"> </span><span class="n">two</span><span class="w"> </span><span class="n">can</span><span class="w"> </span><span class="k">continue</span><span class="o">.</span>
<span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="n">Returns</span><span class="w"> </span><span class="n">from</span><span class="w"> </span><span class="n">wait</span><span class="p">(),</span><span class="w"> </span><span class="n">locks</span><span class="w"> </span><span class="n">mutex</span><span class="o">.</span>
<span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">Unlocks</span><span class="w"> </span><span class="n">mutex</span><span class="o">.</span>
</code></pre></div>
<p>This scenario works too.</p>
<p>There's one change that we should do though. Instead of an if statement in line 7, we should use a while. Why? Because when thread two wakes up in line 7 and obtains a lock on the mutex, it should re-verify that the condition that made it wait is not true anymore. While not a problem in this code segment, this can be a problem in other cases.</p>Pitfalls of POSIX condition variables.2016-11-24T20:23:00-08:002016-11-24T20:23:00-08:00Petko Minkovtag:pminkov.github.io,2016-11-24:/blog/pitfalls-of-posix-condition-variables.html<p>I'm reading this awesome <a href="http://pages.cs.wisc.edu/~remzi/OSTEP/">book</a> about operating systems. I find that with OSs, you can't understand things by just reading the text.
You have to write code, write out things on paper and read certain passages multiple times. That's the difference between learning and reading. Hopefully I'm learning something.</p>
<p>So …</p><p>I'm reading this awesome <a href="http://pages.cs.wisc.edu/~remzi/OSTEP/">book</a> about operating systems. I find that with OSs, you can't understand things by just reading the text.
You have to write code, write out things on paper and read certain passages multiple times. That's the difference between learning and reading. Hopefully I'm learning something.</p>
<p>So this book has a nice chapter on <a href="https://computing.llnl.gov/tutorials/pthreads/#ConditionVariables">condition variables</a>. The way condition variables operate is fairly clear, but I found two pitfalls that I had to dig deeper into in order to see what's going on.</p>
<p>First, a crash course in how condition variables operate. I'm using shortened function names for readability. A condition variable is a variable that you can use to make a thread wait for some condition to be true. So roughly, it looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="o">//</span><span class="w"> </span><span class="n">Thread</span><span class="w"> </span><span class="n">one</span><span class="o">.</span>
<span class="n">lock</span><span class="p">(</span><span class="n">mutex</span><span class="p">);</span>
<span class="n">wait</span><span class="p">(</span><span class="n">condition</span><span class="p">,</span><span class="w"> </span><span class="n">mutex</span><span class="p">);</span>
<span class="n">unlock</span><span class="p">(</span><span class="n">mutex</span><span class="p">);</span>
<span class="o">//</span><span class="w"> </span><span class="n">Thread</span><span class="w"> </span><span class="n">two</span><span class="o">.</span>
<span class="n">lock</span><span class="p">(</span><span class="n">mutex</span><span class="p">);</span>
<span class="k">signal</span><span class="p">(</span><span class="n">condition</span><span class="p">);</span>
<span class="n">unlock</span><span class="p">(</span><span class="n">mutex</span><span class="p">);</span>
</code></pre></div>
<p>Let's say that the first thing that happens here is that thread one obtains the mutex. Thread two runs and blocks on its call to <code>lock</code>. Now thread one proceeds and calls <code>wait(condition, mutex)</code>. What happens here is that mutex is unlocked and thread one goes to sleep. It's waiting for the condition to become true. Thread two is able to obtain the lock and it then signals to thread one that the condition is satisfied. Thread one obtains a lock on the mutex again and finishes its job.</p>
<p>Now that this is out of the way, let's look at the pitfalls.</p>
<h4 id="pitfall-1">Pitfall 1</h4>
<blockquote>
<p>When a thread signals a condition, if the thread holds a lock on the mutex, a waiting thread returns from <code>wait()</code> only after the first thread unlocks the mutex.</p>
</blockquote>
<p>Looking at our example, intuitively it looks like when we call <code>signal()</code>, the first thread should return from wait. After all, the signal has been sent. But that's not the case. If thread two has more work to do after the signal, all of this will be done and just once <code>unlock()</code> is called, that's when thread one will return from the wait.
Thread one will then lock the mutex and once it's done with its work, it will unlock it. This is way more logical than the mess that would happen if <code>signal()</code> makes <code>wait()</code> return immediately.</p>
<h4 id="pitfall-2">Pitfall 2</h4>
<blockquote>
<p>Calling <code>signal()</code> doesn't necessarily unblock all threads waiting on this signal.</p>
</blockquote>
<p>Well this one caused me to scratch my head when I was going through one of the exercises in the book. I assumed a call to <code>signal()</code> simply unblocks all the waiting threads. No, it possibly unblocks only one of them. Apparently there's a <code>pthread_cond_broadcast</code> <a href="https://linux.die.net/man/3/pthread_cond_signal">function</a> that can unblock all waiting threads.</p>
<p>So that's it. Happy coding.</p>Implementing a fast multi-threaded counter.2016-11-23T11:02:00-08:002016-11-23T11:02:00-08:00Petko Minkovtag:pminkov.github.io,2016-11-23:/blog/implementing-a-fast-multi-threaded-counter.html<p>Today I'll write a bit about implementing a simple thread safe counter and improving its speed.</p>
<p>Implementing a basic mutli-threaded counter is a fairly easy task. Using pthreads, you just need to wrap the counter increment in a lock.</p>
<p>The code (<a href="https://github.com/pminkov/wip/tree/master/mt-counters">github link</a>) looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#include</span><span class="w"> </span><span class="cpf">"mythreads.h"</span>
<span class="k">struct …</span></code></pre></div><p>Today I'll write a bit about implementing a simple thread safe counter and improving its speed.</p>
<p>Implementing a basic mutli-threaded counter is a fairly easy task. Using pthreads, you just need to wrap the counter increment in a lock.</p>
<p>The code (<a href="https://github.com/pminkov/wip/tree/master/mt-counters">github link</a>) looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#include</span><span class="w"> </span><span class="cpf">"mythreads.h"</span>
<span class="k">struct</span><span class="w"> </span><span class="nc">counter_t</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">value</span><span class="p">;</span>
<span class="w"> </span><span class="n">pthread_mutex_t</span><span class="w"> </span><span class="n">lock</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">static</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">counter_t</span><span class="w"> </span><span class="n">counter</span><span class="p">;</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">init</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">counter_t</span><span class="w"> </span><span class="o">*</span><span class="n">c</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">c</span><span class="o">-></span><span class="n">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="n">pthread_mutex_init</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="o">-></span><span class="n">lock</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">);</span>
<span class="p">}</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">increment_by</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">counter_t</span><span class="w"> </span><span class="o">*</span><span class="n">c</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">by</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Pthread_mutex_lock</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="o">-></span><span class="n">lock</span><span class="p">);</span>
<span class="w"> </span><span class="n">c</span><span class="o">-></span><span class="n">value</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="n">by</span><span class="p">;</span>
<span class="w"> </span><span class="n">Pthread_mutex_unlock</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="o">-></span><span class="n">lock</span><span class="p">);</span>
<span class="p">}</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">increment</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">counter_t</span><span class="w"> </span><span class="o">*</span><span class="n">c</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">increment_by</span><span class="p">(</span><span class="n">c</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="kt">int</span><span class="w"> </span><span class="nf">get</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">counter_t</span><span class="w"> </span><span class="o">*</span><span class="n">c</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Pthread_mutex_lock</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="o">-></span><span class="n">lock</span><span class="p">);</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">rc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">c</span><span class="o">-></span><span class="n">value</span><span class="p">;</span>
<span class="w"> </span><span class="n">Pthread_mutex_unlock</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="o">-></span><span class="n">lock</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">rc</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>Nothing complicated. Let's say that we want to increment this counter 1,000,000 times. And let's do this with an increasing amount of threads, each thread incrementing the counter 1,000,000 times. I get the following timings for this exercise.</p>
<div class="highlight"><pre><span></span><code><span class="mf">1</span><span class="w"> </span><span class="n">thread</span><span class="p">:</span><span class="w"> </span><span class="mf">0.064</span><span class="n">s</span><span class="w"> </span><span class="n">real</span><span class="w"> </span><span class="n">time</span><span class="mf">.</span>
<span class="mf">2</span><span class="w"> </span><span class="n">threads</span><span class="p">:</span><span class="w"> </span><span class="mf">9.930</span><span class="n">s</span>
<span class="mf">4</span><span class="w"> </span><span class="n">threads</span><span class="p">:</span><span class="w"> </span><span class="mf">23.971</span><span class="n">s</span>
</code></pre></div>
<p>This counter is really slow. Also, it doesn't scale well, since at the moment more than one thread starts to use it, it becomes so much slower. Why is this? Well, it's a bit difficult to tell without knowing how mutexes are implemented, but since we're using a single mutex that has to switch between two threads, it looks like there's a lot of overhead in this. The core operation - the increment, is also not parallel, since it can be done by only one thread at a time. But judging from the single threaded timing, this operation by itself is not the bottleneck here. So the synchronization must be.</p>
<p>How can we improve this? We can use what's called a sloppy counter. The sloppy counter is also fairly easy to understand. Each thread has its own counter and when that counter becomes bigger than a certain value, its current value is transferred into a global counter. Here's how the code for that looks like:</p>
<div class="highlight"><pre><span></span><code><span class="n">#include</span><span class="w"> </span><span class="o"><</span><span class="n">string</span><span class="p">.</span><span class="n">h</span><span class="o">></span>
<span class="n">#include</span><span class="w"> </span><span class="o"><</span><span class="n">stdlib</span><span class="p">.</span><span class="n">h</span><span class="o">></span>
<span class="n">#define</span><span class="w"> </span><span class="nf">min</span><span class="p">(</span><span class="n">a</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="n">a</span><span class="p">)</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="p">(</span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="vm">?</span><span class="w"> </span><span class="p">(</span><span class="n">a</span><span class="p">)</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="p">(</span><span class="n">b</span><span class="p">))</span>
<span class="n">const</span><span class="w"> </span><span class="n">uint64_t</span><span class="w"> </span><span class="n">SLOTS_COUNT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">101</span><span class="p">;</span>
<span class="n">struct</span><span class="w"> </span><span class="n">sloppy_counter_t</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="nc">int</span><span class="w"> </span><span class="k">value</span><span class="p">;</span>
<span class="w"> </span><span class="n">struct</span><span class="w"> </span><span class="n">counter_t</span><span class="w"> </span><span class="n">gcounter</span><span class="p">;</span>
<span class="w"> </span><span class="o">//</span><span class="w"> </span><span class="n">A</span><span class="w"> </span><span class="n">hash</span><span class="w"> </span><span class="nc">table</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">per</span><span class="o">-</span><span class="n">thread</span><span class="w"> </span><span class="n">counters</span><span class="p">.</span><span class="w"> </span><span class="n">Since</span><span class="w"> </span><span class="n">we</span><span class="s1">'re unlikely to run too many threads at the same time,</span>
<span class="s1"> // chances for collision are low. If that'</span><span class="n">s</span><span class="w"> </span><span class="ow">not</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="k">case</span><span class="p">,</span><span class="w"> </span><span class="n">we</span><span class="w"> </span><span class="n">can</span><span class="w"> </span><span class="n">always</span><span class="w"> </span><span class="k">use</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">per</span><span class="o">-</span><span class="n">counter</span><span class="w"> </span><span class="n">mutex</span><span class="p">.</span>
<span class="w"> </span><span class="nc">int</span><span class="w"> </span><span class="n">lcounters</span><span class="o">[</span><span class="n">SLOTS_COUNT</span><span class="o">]</span><span class="p">;</span>
<span class="err">}</span><span class="p">;</span>
<span class="k">static</span><span class="w"> </span><span class="n">struct</span><span class="w"> </span><span class="n">sloppy_counter_t</span><span class="w"> </span><span class="n">sloppy_counter</span><span class="p">;</span>
<span class="n">void</span><span class="w"> </span><span class="n">sloppy_init</span><span class="p">(</span><span class="n">struct</span><span class="w"> </span><span class="n">sloppy_counter_t</span><span class="w"> </span><span class="o">*</span><span class="n">c</span><span class="p">)</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="nc">int</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">SLOTS_COUNT</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">c</span><span class="o">-></span><span class="n">lcounters</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="err">}</span>
<span class="w"> </span><span class="n">init</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="o">-></span><span class="n">gcounter</span><span class="p">);</span>
<span class="err">}</span>
<span class="nc">int</span><span class="w"> </span><span class="n">slot_id</span><span class="p">(</span><span class="n">pthread_t</span><span class="w"> </span><span class="n">thread_id</span><span class="p">)</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">uint64_t</span><span class="w"> </span><span class="n">ptid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="n">memcpy</span><span class="p">(</span><span class="o">&</span><span class="n">ptid</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">thread_id</span><span class="p">,</span><span class="w"> </span><span class="nf">min</span><span class="p">(</span><span class="n">sizeof</span><span class="p">(</span><span class="n">thread_id</span><span class="p">),</span><span class="w"> </span><span class="n">sizeof</span><span class="p">(</span><span class="n">ptid</span><span class="p">)));</span>
<span class="w"> </span><span class="nc">int</span><span class="w"> </span><span class="n">sid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ptid</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="n">SLOTS_COUNT</span><span class="p">;</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">sid</span><span class="p">;</span>
<span class="err">}</span>
<span class="n">void</span><span class="w"> </span><span class="n">sloppy_increment</span><span class="p">(</span><span class="n">struct</span><span class="w"> </span><span class="n">sloppy_counter_t</span><span class="w"> </span><span class="o">*</span><span class="n">c</span><span class="p">,</span><span class="w"> </span><span class="n">pthread_t</span><span class="w"> </span><span class="n">thread_id</span><span class="p">)</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="nc">int</span><span class="w"> </span><span class="n">sid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">slot_id</span><span class="p">(</span><span class="n">thread_id</span><span class="p">);</span>
<span class="w"> </span><span class="n">c</span><span class="o">-></span><span class="n">lcounters</span><span class="o">[</span><span class="n">sid</span><span class="o">]++</span><span class="p">;</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">c</span><span class="o">-></span><span class="n">lcounters</span><span class="o">[</span><span class="n">sid</span><span class="o">]</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">128</span><span class="p">)</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">increment_by</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="o">-></span><span class="n">gcounter</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="o">-></span><span class="n">lcounters</span><span class="o">[</span><span class="n">sid</span><span class="o">]</span><span class="p">);</span>
<span class="w"> </span><span class="n">c</span><span class="o">-></span><span class="n">lcounters</span><span class="o">[</span><span class="n">sid</span><span class="o">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="err">}</span>
<span class="err">}</span>
<span class="n">void</span><span class="w"> </span><span class="n">sloppy_flush</span><span class="p">(</span><span class="n">struct</span><span class="w"> </span><span class="n">sloppy_counter_t</span><span class="w"> </span><span class="o">*</span><span class="n">c</span><span class="p">,</span><span class="w"> </span><span class="n">pthread_t</span><span class="w"> </span><span class="n">thread_id</span><span class="p">)</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="nc">int</span><span class="w"> </span><span class="n">sid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">slot_id</span><span class="p">(</span><span class="n">thread_id</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">c</span><span class="o">-></span><span class="n">lcounters</span><span class="o">[</span><span class="n">sid</span><span class="o">]</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">increment_by</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="o">-></span><span class="n">gcounter</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="o">-></span><span class="n">lcounters</span><span class="o">[</span><span class="n">sid</span><span class="o">]</span><span class="p">);</span>
<span class="w"> </span><span class="err">}</span>
<span class="err">}</span>
<span class="nc">int</span><span class="w"> </span><span class="n">sloppy_get</span><span class="p">(</span><span class="n">struct</span><span class="w"> </span><span class="n">sloppy_counter_t</span><span class="w"> </span><span class="o">*</span><span class="n">counter</span><span class="p">)</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="k">get</span><span class="p">(</span><span class="o">&</span><span class="n">counter</span><span class="o">-></span><span class="n">gcounter</span><span class="p">);</span>
<span class="err">}</span>
</code></pre></div>
<p>Now let's time this counter.</p>
<div class="highlight"><pre><span></span><code><span class="mf">1</span><span class="w"> </span><span class="n">thread</span><span class="p">:</span><span class="w"> </span><span class="mf">0.026</span><span class="n">s</span>
<span class="mf">2</span><span class="w"> </span><span class="n">threads</span><span class="p">:</span><span class="w"> </span><span class="mf">0.050</span><span class="n">s</span>
<span class="mf">4</span><span class="w"> </span><span class="n">threads</span><span class="p">:</span><span class="w"> </span><span class="mf">0.164</span><span class="n">s</span>
</code></pre></div>
<p>Much better! This counter, just like the first one, is thread safe. It's not as accurate, but the inaccuracy is small (at most 128 * number of threads) and we can use the flush function if we want accurate counts.</p>Monitoring Disk I/O on Linux.2016-11-16T14:59:00-08:002016-11-16T14:59:00-08:00Petko Minkovtag:pminkov.github.io,2016-11-16:/blog/monitoring-disk-io-on-linux.html<p>Today I wrote a little piece of code that generates a random array of numbers, stores it into disk and sorts it on disk, using bubble sort. This uses O(1) memory, but it's obviously very slow. I did this for fun mostly. The code is <a href="https://github.com/pminkov/wip/blob/master/os/disksort.c">here</a>.</p>
<p>I wanted to …</p><p>Today I wrote a little piece of code that generates a random array of numbers, stores it into disk and sorts it on disk, using bubble sort. This uses O(1) memory, but it's obviously very slow. I did this for fun mostly. The code is <a href="https://github.com/pminkov/wip/blob/master/os/disksort.c">here</a>.</p>
<p>I wanted to know, if I'm running this code, what Linux tools are going to show an increase in disk I/O.</p>
<p>So, I run this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>./disksort<span class="w"> </span><span class="m">150000</span>
</code></pre></div>
<p>This starts the sorting. It takes a long time to sort 150k numbers on disk, at least with that algorithm.</p>
<p>So the first command that we can run is <code>iostat</code>. Let's see what it is outputting before I start <code>disksort</code></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>iostat<span class="w"> </span><span class="m">5</span><span class="w"> </span><span class="m">2</span>
Linux<span class="w"> </span><span class="m">4</span>.4.0-31-generic<span class="w"> </span><span class="o">(</span>petko-VirtualBox<span class="o">)</span><span class="w"> </span><span class="m">11</span>/16/2016<span class="w"> </span>_x86_64_<span class="w"> </span><span class="o">(</span><span class="m">1</span><span class="w"> </span>CPU<span class="o">)</span>
avg-cpu:<span class="w"> </span>%user<span class="w"> </span>%nice<span class="w"> </span>%system<span class="w"> </span>%iowait<span class="w"> </span>%steal<span class="w"> </span>%idle
<span class="w"> </span><span class="m">14</span>.68<span class="w"> </span><span class="m">0</span>.19<span class="w"> </span><span class="m">38</span>.34<span class="w"> </span><span class="m">0</span>.05<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">46</span>.73
Device:<span class="w"> </span>tps<span class="w"> </span>kB_read/s<span class="w"> </span>kB_wrtn/s<span class="w"> </span>kB_read<span class="w"> </span>kB_wrtn
sda<span class="w"> </span><span class="m">5</span>.60<span class="w"> </span><span class="m">83</span>.17<span class="w"> </span><span class="m">63</span>.96<span class="w"> </span><span class="m">539970</span><span class="w"> </span><span class="m">415252</span>
avg-cpu:<span class="w"> </span>%user<span class="w"> </span>%nice<span class="w"> </span>%system<span class="w"> </span>%iowait<span class="w"> </span>%steal<span class="w"> </span>%idle
<span class="w"> </span><span class="m">3</span>.03<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">96</span>.97
Device:<span class="w"> </span>tps<span class="w"> </span>kB_read/s<span class="w"> </span>kB_wrtn/s<span class="w"> </span>kB_read<span class="w"> </span>kB_wrtn
sda<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
</code></pre></div>
<p>The command line invocation is a bit strange. The <code>5 2</code> part says "aggregate IO data for five seconds and show me two reports". The idea is that you can get continuous report output every five seconds. But I need only one report. The first one is a default one, which shows aggregated data for I'm not sure what period. But the second one is interesting. Notice <code>kB_read/s</code> and <code>kB_wrtn/s</code>. They're zeros. So it's all quiet. I now run <code>disksort</code> and run the same command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>iostat<span class="w"> </span><span class="m">5</span><span class="w"> </span><span class="m">2</span>
Linux<span class="w"> </span><span class="m">4</span>.4.0-31-generic<span class="w"> </span><span class="o">(</span>petko-VirtualBox<span class="o">)</span><span class="w"> </span><span class="m">11</span>/16/2016<span class="w"> </span>_x86_64_<span class="w"> </span><span class="o">(</span><span class="m">1</span><span class="w"> </span>CPU<span class="o">)</span>
avg-cpu:<span class="w"> </span>%user<span class="w"> </span>%nice<span class="w"> </span>%system<span class="w"> </span>%iowait<span class="w"> </span>%steal<span class="w"> </span>%idle
<span class="w"> </span><span class="m">15</span>.32<span class="w"> </span><span class="m">0</span>.17<span class="w"> </span><span class="m">40</span>.33<span class="w"> </span><span class="m">0</span>.05<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">44</span>.14
Device:<span class="w"> </span>tps<span class="w"> </span>kB_read/s<span class="w"> </span>kB_wrtn/s<span class="w"> </span>kB_read<span class="w"> </span>kB_wrtn
sda<span class="w"> </span><span class="m">5</span>.04<span class="w"> </span><span class="m">73</span>.65<span class="w"> </span><span class="m">59</span>.42<span class="w"> </span><span class="m">540254</span><span class="w"> </span><span class="m">435868</span>
avg-cpu:<span class="w"> </span>%user<span class="w"> </span>%nice<span class="w"> </span>%system<span class="w"> </span>%iowait<span class="w"> </span>%steal<span class="w"> </span>%idle
<span class="w"> </span><span class="m">26</span>.81<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">73</span>.19<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">0</span>.00
Device:<span class="w"> </span>tps<span class="w"> </span>kB_read/s<span class="w"> </span>kB_wrtn/s<span class="w"> </span>kB_read<span class="w"> </span>kB_wrtn
sda<span class="w"> </span><span class="m">0</span>.80<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span><span class="m">157</span>.64<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">588</span>
</code></pre></div>
<p>Notice how <code>kB_wrtn/s</code> spiked up. <code>kB_read</code> is zero and I assume that's because of the <a href="http://www.tldp.org/LDP/sag/html/buffer-cache.html">buffer cache</a>. After all, I'm reading the same file again and again.</p>
<p>Another command that we can use is <code>iotop</code>. <code>iotop</code> is similar to <code>top</code>, but shows I/O stats. We'll run it using <code>iotop -oa</code>. The <code>-o</code> parameter makes it show only processes that do I/O. The <code>-a</code> flag aggregates the data during the time this command is running. So after running it for a few seconds I'm seeing this:</p>
<div class="highlight"><pre><span></span><code>Total DISK READ : 0.00 B/s | Total DISK WRITE : 83.77 K/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
545 be/3 root 0.00 B 4.00 K 0.00 % 0.08 % [jbd2/sda1-8]
7717 be/4 root 0.00 B 0.00 B 0.00 % 0.01 % [kworker/u2:3]
7865 be/4 root 0.00 B 0.00 B 0.00 % 0.01 % [kworker/u2:1]
7714 be/4 petko 0.00 B 1144.00 K 0.00 % 0.00 % ./disksort 150000
7097 be/4 root 0.00 B 0.00 B 0.00 % 0.00 % [kworker/u2:2]
7758 be/4 root 0.00 B 0.00 B 0.00 % 0.00 % [kworker/u2:0]
</code></pre></div>
<p>Notice how <code>disksort</code> with pid of <code>7714</code> is well ahead of everything else shown.</p>
<p>So that's it, happy debugging.</p>Using pandas to read a table from an HTML page.2016-10-31T13:56:00-07:002016-10-31T13:56:00-07:00Petko Minkovtag:pminkov.github.io,2016-10-31:/blog/using-pandas-to-read-a-table-from-an-html-page.html<p>Today I wanted to write a bit of simple code to try out a hypothesis I had about stock prices. I found historical data at <a href="http://www.multpl.com/s-p-500-historical-prices/table/by-year">multpl.com</a>. At first I thought I'd have to write my own code using Python's <a href="https://docs.python.org/2/library/htmlparser.html">HTMLParser</a>. As much as I like to write code, I …</p><p>Today I wanted to write a bit of simple code to try out a hypothesis I had about stock prices. I found historical data at <a href="http://www.multpl.com/s-p-500-historical-prices/table/by-year">multpl.com</a>. At first I thought I'd have to write my own code using Python's <a href="https://docs.python.org/2/library/htmlparser.html">HTMLParser</a>. As much as I like to write code, I decided to save myself some time and find something that already does this. To my surprise, <a href="http://pandas.pydata.org/">pandas</a>, already has a <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_html.html">function</a> that reads data from HTML tables. Great. Let's see how it works.</p>
<p>First, I installed pandas, lxml and a bunch of other requirements in a <a href="http://pminkov.github.io/why-i-always-use-virtualenv-to-try-out-new-libraries.html">virtual environment</a>.</p>
<p>To read the tables from the webpage, I used the following lines of code:</p>
<div class="highlight"><pre><span></span><code> <span class="n">tables</span> <span class="o">=</span> <span class="n">pandas</span><span class="o">.</span><span class="n">read_html</span><span class="p">(</span><span class="s1">'http://www.multpl.com/s-p-500-historical-prices/table/by-month'</span><span class="p">,</span> <span class="n">header</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">tables</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span>
<span class="n">table</span> <span class="o">=</span> <span class="n">tables</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</code></pre></div>
<p>The <code>header</code> parameter, says that we should use the first row in a table as the dataframe's column names. Spefically for this table, one of the columns was getting a somewhat wrong name, but it was easy to fix it with this line of code:</p>
<div class="highlight"><pre><span></span><code><span class="n">table</span><span class="o">.</span><span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'Date'</span><span class="p">,</span> <span class="s1">'Price'</span><span class="p">]</span>
</code></pre></div>
<p>There we go, we have read historical stock prices into a pandas dataframe. To iterate and print them, we can do it like this:</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span> <span class="p">(</span><span class="n">idx</span><span class="p">,</span> <span class="n">row</span><span class="p">)</span> <span class="ow">in</span> <span class="n">table</span><span class="o">.</span><span class="n">iterrows</span><span class="p">():</span>
<span class="n">date</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="s1">'Date'</span><span class="p">]</span>
<span class="n">price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="s1">'Price'</span><span class="p">]</span>
<span class="nb">print</span> <span class="n">date</span><span class="p">,</span> <span class="n">price</span>
</code></pre></div>What I learned from reading Dive into Python.2016-06-15T09:57:00-07:002016-06-15T09:57:00-07:00Petko Minkovtag:pminkov.github.io,2016-06-15:/blog/what-i-learned-from-reading-dive-into-python.html<p>I recently started reading <a href="http://www.diveintopython.net/toc/index.html">Dive into Python</a>. I've been meaning to fill some gaps in my Python knowledge, since I've used the language for about ten years, but feel like I have just picked whatever parts I needed to do my work and don't have a very solid base.</p>
<p>I'll …</p><p>I recently started reading <a href="http://www.diveintopython.net/toc/index.html">Dive into Python</a>. I've been meaning to fill some gaps in my Python knowledge, since I've used the language for about ten years, but feel like I have just picked whatever parts I needed to do my work and don't have a very solid base.</p>
<p>I'll summarize some of the new things I learned from this book:</p>
<h3 id="chapter-2-your-first-python-program">Chapter 2 (Your First Python Program)</h3>
<ul>
<li><code>sys.path</code> contains the list of directories that Python uses to lookup module imports.</li>
<li>One use of the <code>__name__</code> attribute is to write testing code. When a module is imported <code>__name__</code> is the name of the module. When a module python file is executed from the command line <code>__name__</code> is equal to <code>__main__</code>. Personally this feels like a bit of a hack to me, since you can use something like say Django's unit testing platform, but it might come handy sometime.</li>
</ul>
<h3 id="chapter-3-native-datatypes">Chapter 3 (Native Datatypes)</h3>
<ul>
<li>Here's a cool trick mentioned in this chapter:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="n">MONDAY</span><span class="p">,</span> <span class="n">TUESDAY</span><span class="p">,</span> <span class="n">WEDNESDAY</span><span class="p">,</span> <span class="n">THURSDAY</span><span class="p">,</span> <span class="n">FRIDAY</span><span class="p">,</span> <span class="n">SATURDAY</span><span class="p">,</span> <span class="n">SUNDAY</span><span class="p">)</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="mi">7</span><span class="p">)</span>
</code></pre></div>
<h3 id="chapter-4-the-power-of-introspection">Chapter 4 (The Power Of Introspection):</h3>
<ul>
<li>Some useful functions are mentioned: <code>dir</code> (try on a module or an object instance), <code>callable</code>, <code>getattr</code>, <code>__doc__</code>.</li>
</ul>
<p>So you can do something interesting, like this:</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="n">Sum:</span>
<span class="s">"""Sums numbers"""</span>
<span class="n">member</span> = <span class="mi">3</span>
<span class="n">def</span> <span class="n">sum2</span>(<span class="nb">self</span>, <span class="n">a</span>, <span class="n">b</span>):
<span class="s">"""Sum two numbers"""</span>
<span class="k">return</span> <span class="n">a</span> + <span class="n">b</span>
<span class="n">def</span> <span class="n">sum3</span>(<span class="nb">self</span>, <span class="n">a</span>, <span class="n">b</span>, <span class="n">c</span>):
<span class="s">"""Sum three numbers"""</span>
<span class="k">return</span> <span class="n">a</span> + <span class="n">b</span> + <span class="n">c</span>
<span class="k">if</span> <span class="n">__name__</span> == <span class="s">'__main__'</span>:
<span class="nb">methods</span> = [<span class="nb">f</span> <span class="k">for</span> <span class="nb">f</span> <span class="nb">in</span> <span class="nb">dir</span>(<span class="n">Sum</span>) <span class="k">if</span> <span class="n">callable</span>(<span class="n">getattr</span>(<span class="n">Sum</span>, <span class="nb">f</span>))]
<span class="n">docs</span> = <span class="s">'\n'</span>.<span class="nb">join</span>([<span class="s">"%s %s"</span> % (<span class="nb">f</span>, <span class="n">getattr</span>(<span class="n">Sum</span>, <span class="nb">f</span>).<span class="n">__doc__</span>) <span class="k">for</span> <span class="nb">f</span> <span class="nb">in</span> <span class="nb">methods</span>])
<span class="nb">print</span> <span class="n">docs</span>
</code></pre></div>
<p>And run it:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>python<span class="w"> </span>./sum.py
sum2<span class="w"> </span>Sum<span class="w"> </span>two<span class="w"> </span>numbers
sum3<span class="w"> </span>Sum<span class="w"> </span>three<span class="w"> </span>numbers
</code></pre></div>
<ul>
<li>The "and-or trick"</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">is_mammal</span> <span class="o">=</span> <span class="n">use_rpc</span> <span class="ow">and</span> <span class="n">remote_check_is_mammal</span><span class="p">(</span><span class="n">animal</span><span class="p">)</span> <span class="ow">or</span> <span class="n">DEFAULT_IS_MAMMAL</span>
</code></pre></div>
<p>That's like <code>?:</code> in C++. I think I did well with creating a good example here :).</p>
<h3 id="chapter-5-objects-and-object-orientation">Chapter 5 (Objects and Object-Orientation)</h3>
<ul>
<li><code>var1 is var2</code> checks for object identity. Here's a good <a href="http://stackoverflow.com/questions/13650293/understanding-pythons-is-operator">stackoverflow</a> article that gives examples.</li>
<li>You can change a class variable using <code>self.__class__.variable_name = new_value</code>.</li>
<li>You can create your own dictionary by inheriting from <code>UserDict</code>.</li>
<li>If you call <code>x["hello"] = 3</code>, this calls the <code>__setitem__</code> method. There are other similar methods, like <code>__getitem__</code>, etc.</li>
</ul>
<h3 id="chapter-6-exceptions-and-file-handling">Chapter 6 (Exceptions and file handling)</h3>
<ul>
<li>You can use <code>else</code> in exceptions code. Like this:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">RPC</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">remote_call</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">find_function</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">a</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">RPC</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
<span class="nb">print</span> <span class="s1">'No such function.'</span>
<span class="k">else</span><span class="p">:</span>
<span class="nb">print</span> <span class="s1">'Function was found.'</span>
<span class="k">finally</span><span class="p">:</span>
<span class="nb">print</span> <span class="s1">'Enough lookups.'</span>
<span class="nb">print</span>
<span class="n">find_function</span><span class="p">(</span><span class="s1">'func'</span><span class="p">)</span>
<span class="n">find_function</span><span class="p">(</span><span class="s1">'remote_call'</span><span class="p">)</span>
</code></pre></div>
<p>Output:</p>
<div class="highlight"><pre><span></span><code>python<span class="w"> </span>./ex.py<span class="w"> </span>
No<span class="w"> </span>such<span class="w"> </span><span class="k">function</span>.
Enough<span class="w"> </span>lookups.
Function<span class="w"> </span>was<span class="w"> </span>found.
Enough<span class="w"> </span>lookups.
</code></pre></div>
<ul>
<li>When you open a file, you have a variety of functions and attributes on the file object, like <code>seek</code>, <code>read</code>, <code>mode</code>, <code>name</code>. For example, to see how big a file is you can just call <code>f.seek(0, 2)</code> to seek until the end and then <code>f.tell()</code> to output the number of bytes.</li>
<li>A class' module is accessible by calling <code>ClassName.__module__</code>. All imported modules are in the <code>sys.modules</code> dictionary. Example:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="kn">from</span> <span class="nn">sum</span> <span class="kn">import</span> <span class="n">Sum</span>
<span class="o">>>></span> <span class="kn">import</span> <span class="nn">sys</span>
<span class="o">>>></span> <span class="n">sys</span><span class="o">.</span><span class="n">modules</span><span class="p">[</span><span class="n">Sum</span><span class="o">.</span><span class="vm">__module__</span><span class="p">]</span>
<span class="o"><</span><span class="n">module</span> <span class="s1">'sum'</span> <span class="kn">from</span> <span class="s1">'sum.py'</span><span class="o">></span>
<span class="o">>>></span> <span class="nb">getattr</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">modules</span><span class="p">[</span><span class="n">Sum</span><span class="o">.</span><span class="vm">__module__</span><span class="p">],</span> <span class="s1">'Sum'</span><span class="p">)</span>
<span class="o"><</span><span class="k">class</span> <span class="nc">sum</span><span class="o">.</span><span class="n">Sum</span> <span class="n">at</span> <span class="mh">0x109535a10</span><span class="o">></span>
</code></pre></div>
<h3 id="chapter-7-regular-expressions">Chapter 7 (Regular Expressions)</h3>
<ul>
<li><code>\\b</code> matches a word boundary. This can be so useful.</li>
<li>Verbose regular expressions. Again, very useful. Example:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">re</span>
<span class="n">raw_regex</span> <span class="o">=</span> <span class="sa">r</span><span class="s2">"""</span>
<span class="s2"> (\d</span><span class="si">{3}</span><span class="s2">)</span>
<span class="s2"> \D*</span>
<span class="s2"> (\d</span><span class="si">{3}</span><span class="s2">)</span>
<span class="s2"> \D*</span>
<span class="s2"> (\d</span><span class="si">{4}</span><span class="s2">)</span>
<span class="s2">"""</span>
<span class="n">examples</span> <span class="o">=</span> <span class="p">[</span>
<span class="s2">"4153125633"</span><span class="p">,</span>
<span class="s2">"415-312-5633"</span><span class="p">,</span>
<span class="s2">"415 312 5633"</span><span class="p">,</span>
<span class="s2">"work 415 312 5633"</span><span class="p">,</span>
<span class="s2">"(415) - 312 - 5633"</span><span class="p">,</span>
<span class="s2">"1 415 312 5633"</span><span class="p">,</span>
<span class="p">]</span>
<span class="n">phone_re</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">raw_regex</span><span class="p">,</span> <span class="n">re</span><span class="o">.</span><span class="n">VERBOSE</span><span class="p">)</span>
<span class="k">for</span> <span class="n">example</span> <span class="ow">in</span> <span class="n">examples</span><span class="p">:</span>
<span class="n">groups</span> <span class="o">=</span> <span class="n">phone_re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="n">example</span><span class="p">)</span><span class="o">.</span><span class="n">groups</span><span class="p">()</span>
<span class="k">assert</span> <span class="n">groups</span> <span class="o">==</span> <span class="p">(</span><span class="s1">'415'</span><span class="p">,</span> <span class="s1">'312'</span><span class="p">,</span> <span class="s1">'5633'</span><span class="p">)</span>
</code></pre></div>Why I always use virtualenv to try out new libraries.2016-05-24T18:14:00-07:002016-05-24T18:14:00-07:00Petko Minkovtag:pminkov.github.io,2016-05-24:/blog/why-i-always-use-virtualenv-to-try-out-new-libraries.html<p>One thing that I always do with virtualenv is to install new Python libraries in a new virtualenv before I install them into the one I'm currently working with.</p>
<p>Here's how I do this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>virtualenv<span class="w"> </span>venv
New<span class="w"> </span>python<span class="w"> </span>executable<span class="w"> </span><span class="k">in</span><span class="w"> </span>/Users/petko/work/post/venv/bin/python
Installing<span class="w"> </span>setuptools,<span class="w"> </span>pip …</code></pre></div><p>One thing that I always do with virtualenv is to install new Python libraries in a new virtualenv before I install them into the one I'm currently working with.</p>
<p>Here's how I do this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>virtualenv<span class="w"> </span>venv
New<span class="w"> </span>python<span class="w"> </span>executable<span class="w"> </span><span class="k">in</span><span class="w"> </span>/Users/petko/work/post/venv/bin/python
Installing<span class="w"> </span>setuptools,<span class="w"> </span>pip,<span class="w"> </span>wheel...done.
$<span class="w"> </span>.<span class="w"> </span>./venv/bin/activate
<span class="o">(</span>venv<span class="o">)</span><span class="w"> </span>
$<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>cryptography
...<span class="w"> </span>lengthy<span class="w"> </span>install<span class="w"> </span>log<span class="w"> </span>...
$<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>pipdeptree<span class="w"> </span><span class="c1"># Awesome tool, more in a bit.</span>
$<span class="w"> </span>pipdeptree
Warning!!!<span class="w"> </span>Possibly<span class="w"> </span>conflicting<span class="w"> </span>dependencies<span class="w"> </span>found:
*<span class="w"> </span><span class="nv">cryptography</span><span class="o">==</span><span class="m">1</span>.3.2
<span class="w"> </span>-<span class="w"> </span>setuptools<span class="w"> </span><span class="o">[</span>required:<span class="w"> </span>><span class="o">=</span><span class="m">11</span>.3,<span class="w"> </span>installed:<span class="w"> </span><unknown><span class="o">]</span>
------------------------------------------------------------------------
<span class="nv">wheel</span><span class="o">==</span><span class="m">0</span>.26.0
<span class="nv">wsgiref</span><span class="o">==</span><span class="m">0</span>.1.2
<span class="nv">cryptography</span><span class="o">==</span><span class="m">1</span>.3.2
<span class="w"> </span>-<span class="w"> </span>setuptools<span class="w"> </span><span class="o">[</span>required:<span class="w"> </span>><span class="o">=</span><span class="m">11</span>.3<span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>enum34<span class="w"> </span><span class="o">[</span>installed:<span class="w"> </span><span class="m">1</span>.1.6<span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>ipaddress<span class="w"> </span><span class="o">[</span>installed:<span class="w"> </span><span class="m">1</span>.0.16<span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>pyasn1<span class="w"> </span><span class="o">[</span>required:<span class="w"> </span>><span class="o">=</span><span class="m">0</span>.1.8,<span class="w"> </span>installed:<span class="w"> </span><span class="m">0</span>.1.9<span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>six<span class="w"> </span><span class="o">[</span>required:<span class="w"> </span>><span class="o">=</span><span class="m">1</span>.4.1,<span class="w"> </span>installed:<span class="w"> </span><span class="m">1</span>.10.0<span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>idna<span class="w"> </span><span class="o">[</span>required:<span class="w"> </span>><span class="o">=</span><span class="m">2</span>.0,<span class="w"> </span>installed:<span class="w"> </span><span class="m">2</span>.1<span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>cffi<span class="w"> </span><span class="o">[</span>required:<span class="w"> </span>><span class="o">=</span><span class="m">1</span>.4.1,<span class="w"> </span>installed:<span class="w"> </span><span class="m">1</span>.6.0<span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>pycparser<span class="w"> </span><span class="o">[</span>installed:<span class="w"> </span><span class="m">2</span>.14<span class="o">]</span>
<span class="o">(</span>venv<span class="o">)</span><span class="w"> </span>
</code></pre></div>
<p>That's it! Now I can experiment with a new library without worrying that it'll pollute my work virtualenv. Another thing that I do is using pipdeptree to see the dependencies that a new library will bring with it. If a library brings in too many dependencies, I'll be thinking twice about my need to use it, or I'll look for alternatives.</p>Using mustache templates with Django.2016-05-23T22:13:00-07:002016-05-23T22:13:00-07:00Petko Minkovtag:pminkov.github.io,2016-05-23:/blog/using-mustache-templates-with-django.html<p>There are various reasons why you might want to use client side templates in a Django app.
For example, you might want to make your initial page load faster, return some of your data in json format and postpone some template rendering until the user needs it.</p>
<p>Let's see how …</p><p>There are various reasons why you might want to use client side templates in a Django app.
For example, you might want to make your initial page load faster, return some of your data in json format and postpone some template rendering until the user needs it.</p>
<p>Let's see how we can easily include <a href="https://mustache.github.io/">mustache</a> templates in Django. <a href="http://mustache.github.io/mustache.5.html">This</a> page explains the mustache syntax quite well.</p>
<p>The most optimized way to include a client side template in your code would be to compile it on deployment and include the compiled JavaScript file as part of your static files.
However, that seemed like too much work and I wanted something that I can implement faster.</p>
<p>If you're not shipping compiled JavaScript, then your other option would be to have the template text somewhere and compile it client side. I have one main requirement though - I don't want my template code to be in a string which editors will display without HTML syntax highlighting.</p>
<p>So, working with these constraints, let's see what we can come up with.</p>
<p>First, let's say our template code is in <code>person.html</code> and it looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">div</span><span class="p">></span>
{{#person}}
The user's name is {{name}}.
{{/person}}
{{^person}}
There's no user defined.
{{/person}}
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
</code></pre></div>
<p>Now, we can include this template in the html page returned by our initial page load, like this:</p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">script</span><span class="p">></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">person_template_text</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sb">`{% include "myapp/templates/person.html" %}`</span>
<span class="p"></</span><span class="nt">script</span><span class="p">></span>
</code></pre></div>
<p>In this case, <code>person_template_text</code> is a <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals">template literal</a>. Template literals are JavaScript strings that can span multiple lines.</p>
<p>Now, we just need to compile the template so that it's ready to be rendered. We'll use <a href="http://twitter.github.io/hogan.js/">hogan.js</a>, since it's advertised as faster and smaller than mustache and it fully supports mustache syntax. Let's say we're also using jQuery. We can do something along these lines:</p>
<div class="highlight"><pre><span></span><code><span class="kd">function</span><span class="w"> </span><span class="nx">App</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">person_template</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">Hogan</span><span class="p">.</span><span class="nx">compile</span><span class="p">(</span><span class="nx">person_template_text</span><span class="p">);</span>
<span class="p">}</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">renderPerson</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kd">function</span><span class="p">(</span><span class="nx">personDict</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">person_template</span><span class="p">.</span><span class="nx">render</span><span class="p">(</span><span class="nx">personDict</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">app</span><span class="p">;</span>
<span class="nx">$</span><span class="p">(</span><span class="nb">document</span><span class="p">).</span><span class="nx">ready</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">app</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">App</span><span class="p">();</span>
<span class="p">});</span>
</code></pre></div>
<p>And that's it. There are many other ways to do this, which will be more optimized, but this one seems like an easy one to start with.</p>Type checking in Python?2016-05-18T10:39:00-07:002016-05-18T10:39:00-07:00Petko Minkovtag:pminkov.github.io,2016-05-18:/blog/type-checking-in-python.html<p>Since Python is a dynamically typed language, it doesn't offer type checking out of the box. There are workarounds around this though.
One that I use in my projects is the following:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">T</span><span class="p">(</span><span class="n">val</span><span class="p">,</span> <span class="n">t</span><span class="p">):</span>
<span class="k">assert</span> <span class="nb">type</span><span class="p">(</span><span class="n">val</span><span class="p">)</span> <span class="o">==</span> <span class="n">t</span><span class="p">,</span> <span class="s1">'(</span><span class="si">%s</span><span class="s1">) </span><span class="si">%s</span><span class="s1"> != </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">val</span><span class="p">),</span> <span class="nb">type</span><span class="p">(</span><span class="n">val</span><span class="p">),</span> <span class="n">t</span><span class="p">)</span>
</code></pre></div>
<p>Here's an …</p><p>Since Python is a dynamically typed language, it doesn't offer type checking out of the box. There are workarounds around this though.
One that I use in my projects is the following:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">T</span><span class="p">(</span><span class="n">val</span><span class="p">,</span> <span class="n">t</span><span class="p">):</span>
<span class="k">assert</span> <span class="nb">type</span><span class="p">(</span><span class="n">val</span><span class="p">)</span> <span class="o">==</span> <span class="n">t</span><span class="p">,</span> <span class="s1">'(</span><span class="si">%s</span><span class="s1">) </span><span class="si">%s</span><span class="s1"> != </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">val</span><span class="p">),</span> <span class="nb">type</span><span class="p">(</span><span class="n">val</span><span class="p">),</span> <span class="n">t</span><span class="p">)</span>
</code></pre></div>
<p>Here's an example of how to use this:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">sum</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
<span class="n">T</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="nb">int</span><span class="p">)</span>
<span class="n">T</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="nb">int</span><span class="p">)</span>
<span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
</code></pre></div>
<p>Let's say that you call the function incorrectly, like this:</p>
<div class="highlight"><pre><span></span><code><span class="n">a</span> <span class="o">=</span> <span class="s2">"hello"</span>
<span class="n">b</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">s</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
</code></pre></div>
<p>You'll get the following exception:</p>
<div class="highlight"><pre><span></span><code>>>> sum(a, b)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in sum
File "<stdin>", line 2, in T
AssertionError: (hello) <type 'str'> != <type 'int'>
</code></pre></div>
<p>So that's it - pretty simple. I use the <code>T</code> function every once in a while, to guard against type errors.</p>How much does it cost to run a Django app on AWS using Elastic Beanstalk?2016-04-20T13:56:00-07:002016-04-20T13:56:00-07:00Petko Minkovtag:pminkov.github.io,2016-04-20:/blog/how-much-does-it-cost-to-run-a-django-app-on-aws-using-elastic-beanstalk.html<p>I've been running a Django app on <a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-django.html">Elastic Beanstalk</a> for a couple of months and I have a decent idea now of the costs involved and the pros and cons of this approach. My goal was to get something going as soon as possible and I'd say Elastic Beanstalk is …</p><p>I've been running a Django app on <a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-django.html">Elastic Beanstalk</a> for a couple of months and I have a decent idea now of the costs involved and the pros and cons of this approach. My goal was to get something going as soon as possible and I'd say Elastic Beanstalk is good for that purpose. There are a few things that took me more time to figure out and I might write about them too, but overall everything is running smoothly now.</p>
<p>Let's first describe what my setup is. Elastic Beanstalk is AWS's PaaS offering. I use a MySQL database running on RDS, a load balancer, a single EC2 instance and I have a DNS setup on Route 53. And that's more or less what I'm paying for. My bill was <strong>$42.58</strong> last month. Breaking it down, here are the three major components it has:</p>
<ul>
<li><strong>EC2</strong></li>
</ul>
<p>I'm running a <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html">t2.micro</a> instance. It costs $0.013 per hour and I paid for 745 hours. Total: <strong>$9.69</strong>.</p>
<ul>
<li><strong>Load balancing</strong></li>
</ul>
<p>Load balancing is not cheap. It costs $0.025 per hour and for 744 hours shown, that comes down to <strong>$18.60</strong>.</p>
<ul>
<li><strong>RDS</strong></li>
</ul>
<p>RDS costs $0.017 per RDS T2 Micro instance hour and for 743 hours shown, I paid <strong>$12.63</strong>.</p>
<p>If we sum these three, it comes down to <strong>$40.92</strong>. I also paid the following:</p>
<ul>
<li>$0.56 for RDS service storage.</li>
<li>$0.45 for <a href="https://aws.amazon.com/ebs/">EBS</a>.</li>
<li>$0.51 for Route 53.</li>
<li>$0.07 for data transfer (my site being served to places around the world, but my site is not popular yet).</li>
<li>$0.05 for S3 costs.</li>
<li>$0.01 for data processed by the load balancer.</li>
<li>$0.01 for SES.</li>
</ul>
<p>These are minimal costs. Some of these are going to increase if my site becomes popular, but right now they are minimal.</p>
<p><strong>How can I bring these costs down?</strong></p>
<p>It's possible to do all of this with a single EC2 instance and avoid paying for load balancing and RDS. What I get from these components right now is convenience and scalability. I don't really need scalability, since I don't operate at scale and might not come to that point. Convenience can be traded for the learning experience of settings up things manually. AWS has an <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-LAMP.html">article</a> on how to install a LAMP setup on EC2. It doesn't look too complicated. Supposedly my bill will be around $10 per month for my 1GB RAM t2.micro machine.</p>
<p>Another option would be to use <a href="https://www.digitalocean.com/pricing/">Digital Ocean</a>. They're basically offering something similar to EC2 instances, but I haven't looked too much in detail. Digital Ocean is an IaaS provider, you need to do some manual setup. Their cost is pretty similar. An offering with 1GB of RAM costs $10 per month too. That's a very rough comparison, but it seems like we're in the same ballpark if we're not pushing the limits on disk space or bandwidth.</p>
<p>It should also be possibe to continue to use Elastic Beanstalk, but run a MySQL server directly on the EC2 instance. Here's a long <a href="http://d0.awsstatic.com/whitepapers/rdbms-in-the-cloud-sql-server-on-aws.pdf">white paper</a> that talks about this.</p>How to generate user activation links in Django.2016-04-05T17:03:00-07:002016-04-05T17:03:00-07:00Petko Minkovtag:pminkov.github.io,2016-04-05:/blog/how-to-generate-user-activation-links-in-django.html<p>Imagine the following simple user registration flow:</p>
<ol>
<li>User registers by setting an e-mail and a password.</li>
<li>The server sends an e-mail containing an activation link to the user.</li>
<li>The user clicks on the activation link and activates their account.</li>
</ol>
<p>Let's focus on step 2. How do we generate this activation …</p><p>Imagine the following simple user registration flow:</p>
<ol>
<li>User registers by setting an e-mail and a password.</li>
<li>The server sends an e-mail containing an activation link to the user.</li>
<li>The user clicks on the activation link and activates their account.</li>
</ol>
<p>Let's focus on step 2. How do we generate this activation link?</p>
<p>One way to implement this is to generate single use tokens and save them in a database table. Our url then looks like <code>http://sitename.com/activate?token=mytoken</code>. That solution has the benefit of being able to track that a token has been used and enforce single use. The drawback is that you need a new database table, which I wanted to avoid.</p>
<p>How else can we generate these tokens?</p>
<p>My first hunch was to reuse Django's <a href="https://github.com/django/django/blob/master/django/contrib/auth/tokens.py">PasswordResetTokenGenerator</a>. However, notice that the <code>check_token</code> function requires a user. This means that the activation link needs to include information about who the user is. So our <code>/activate</code> link will need to include an url parameter that's either the username, user id, or something else that uniquely identifies a user. I'm not happy with that. This is a link that's supposed to be private and I don't want the user's first impression of my site to be that I'm including their user id in an activation link, when most other sites don't do that.</p>
<p>The solution that I settled on was encrypting the user id on the server. I'm generating a link of the form <code>http://sitename.com/activate?token=mytoken</code>. I generate the token using the Python <a href="https://pypi.python.org/pypi/cryptography">cryptography</a> package. The token is created by encrypting the username with a salt.</p>
<p>So why is this not what Django is doing? I suppose the answer is that Python doesn't have a symmetric encryption implementation library coming with it.</p>
<p>In addition, I also wanted the tokens to expire. This is simple, we can just include a timestamp in the token and compare it against the current time upon decryption.</p>
<p>Here's a sample class that implements all of this:</p>
<script src="https://gist.github.com/pminkov/e53d90c348f1dc47553408666431d2a2.js"></script>Plotting in Octave.2012-07-29T22:33:00-07:002012-07-29T22:33:00-07:00softwarecommentstag:pminkov.github.io,2012-07-29:/blog/plotting-in-octave.html<p>Plotting functions in Octave is quite easy and helpful.</p>
<p></p>
In the lecture on neural networks from Stanford's
<a href="http://ml-class.org/">machine learning class</a> a function related to neural
networks is introduced. The function has two inputs - x1 and x2 and it's
supposed to model a logical AND operation. The function is based on …<p>Plotting functions in Octave is quite easy and helpful.</p>
<p></p>
In the lecture on neural networks from Stanford's
<a href="http://ml-class.org/">machine learning class</a> a function related to neural
networks is introduced. The function has two inputs - x1 and x2 and it's
supposed to model a logical AND operation. The function is based on the
sigmoid function.</p>
<p></p>
The sigmoid function looks like this:</p>
<p><center><img alt="Google sigmoid plot" src="https://pminkov.github.io/blog/images/sigmoid-google.png"></center></p>
<p></p>
Computing the sigmoid function in Octave can be seen
<a href="https://gist.github.com/3205075" title="Sidmoid">here</a>.</p>
<p></p>
The given example plugs x1 and x2 into the sigmoid function by
computing:</p>
<p></p>
f(x1, x2) = sigmoid(-30 + 20*x1 + 20*x2)</p>
<p></p>
f(1, 1) evaluates to 1</p>
<p></p>
f(0, 0), f(0, 1) and f(1, 0) evaluate to 0.</p>
<p></p>
Intuitively that make sense. Even if x1 or x2 is 1, -30 + 20 is -10,
which corresponds to a very low value for the sigmoid function, as you
can see from the graph above.</p>
<p></p>
I got curious about how this function looks like for values between zero
and one and wrote this <a href="https://gist.github.com/3205016">short Octave
program</a> to plot the output. And here
it is. Pretty neat, I'd say.</p>
<p><center><img alt="Octave sigmoid plot" src="https://pminkov.github.io/blog/images/sigmoid-octave.png"></center></p>
<p></p>
It's easy to see that the function behaves as expected at (0,0), (0,1)
and (1,0) and (1,1) and that for values in between its computed values
are somewhere in between as well.</p>
</p>Stanford's machine learning class.2012-07-27T15:00:45-07:002012-07-27T15:00:45-07:00Petko Minkovtag:pminkov.github.io,2012-07-27:/blog/stanfords-machine-learning-class.html<p>Stanford's machine learning class on coursera is pretty good. What's
surprising to me is that exercises take much more time than listening to
lectures and taking after class quizes.</p>
<p></p>
The programming environment used for the class is Octave. The most
challenging part so far with writing code in Octave is …<p>Stanford's machine learning class on coursera is pretty good. What's
surprising to me is that exercises take much more time than listening to
lectures and taking after class quizes.</p>
<p></p>
The programming environment used for the class is Octave. The most
challenging part so far with writing code in Octave is coming up with
what's called vectorization solutions. The idea of vectorization is that
instead of writing loops, you compute values by vector and matrix
operations.</p>
<p></p>
Let's look at a simple example. Imagine that you have a vector of zeros
and ones. You have to convert it to a vector where for each one we have
a one and for each zero we have minus one. That came up as a component
of the cost function computation in logistic regression.</p>
<p></p>
An iterative solution in Scala might look like this:</p>
<p></p>
v.map(x => if (x == 0) -1 else 1)</p>
<p></p>
How would that look in Octave if we can't use loops and if statements? A
function F that maps from {0, 1} to {-1, 1} actually looks like this:</p>
<p></p>
F(x) = 2x - 1</p>
<p></p>
Pretty cool. Now if our vector of zeros and ones is A, by using that
function, we can compute the desired value by doing this:</p>
<p></p>
D = 2 * A - 1</p>
<p></p>
This is pretty much only a part of a typical vectorized computation in
machine learning. Crafting those takes some time, but at the end there's
a good a-ha moment and usually a quite short solution.</p>
<p></p>
So far I went through the linear and logistic regression lectures. Both
methods are widely practical and it's very easy to come up with usage
example. Those methods are examples of supervised learning algorithms,
since we train them on a data set and then once we've computed a model
we just plug new examples into it and come up with results.</p>
<p></p>
In practice, I doubt that anyone is going to implement their own
gradient descent algorithms to come up with model coefficients. I'd
expect that usually a library / package like Matlab / Octave / R will be
used to train a model. Implementing the model itself is quite trivial
and would be just a couple of lines of code.</p>
</p>Scala scopes.2012-04-27T23:26:00-07:002012-04-27T23:26:00-07:00softwarecommentstag:pminkov.github.io,2012-04-27:/blog/scala-scopes.html<p>After some programming with Scala, there are a couple of language
features which I really like. One of them is scopes that can evaluate to
a value. How does that look like:</p>
<p>
<script src="https://gist.github.com/2516506.js" type="text/javascript"><![CDATA[// <![CDATA[</p><p>// <![CDATA[</p><p> </p><p>// ]]]]]]><![CDATA[><![CDATA[></p><p>// ]]]]><![CDATA[>]]></script>
</p>
<p></p>
Ignore the obvious inefficiencies …<p>After some programming with Scala, there are a couple of language
features which I really like. One of them is scopes that can evaluate to
a value. How does that look like:</p>
<p>
<script src="https://gist.github.com/2516506.js" type="text/javascript"><![CDATA[// <![CDATA[</p><p>// <![CDATA[</p><p> </p><p>// ]]]]]]><![CDATA[><![CDATA[></p><p>// ]]]]><![CDATA[>]]></script>
</p>
<p></p>
Ignore the obvious inefficiencies of the code and take a look at how you
can put a block of code inside the inner scope. What's good about that?
This inner scope is like a lighweight function. The benefits are:</p>
</p>
<ul>
<li>
<p>Variables in the function are not seen outside of the scope. Great,
because you'll get a compile error if you try to use one of
those variables. Once you're working in the scope you know that any
variable you declare inside of it won't be used on the outside.</p>
</li>
<li>
<p>You have a logical block of computation that has a clear start and
finish, which makes the code easier to read and work with.</p>
</li>
<li>
<p>Unlike a function, you don't need to declare function parameters.</p>
</li>
</ul>
<p>The drawback is of course that this can become an excuse for creating
long functions, with a lot of scopes like this one inside of them. If
that's avoided, this language feature leads to easier to understand
code.</p>
</p>Fun problem with Scala.2012-03-06T23:28:00-08:002012-03-06T23:28:00-08:00Petko Minkovtag:pminkov.github.io,2012-03-06:/blog/fun-problem-with-scala.html<p>Here's a fun problem to solve with Scala.</p>
<p></p>
Given N lists, dedupe them with the following rules. Go through the
first list, dedupe it. Go through every consecutive list and only leave
items which are not in a previous list and also take care of removing
duplicates.</p>
<p></p>
The challenge here …<p>Here's a fun problem to solve with Scala.</p>
<p></p>
Given N lists, dedupe them with the following rules. Go through the
first list, dedupe it. Go through every consecutive list and only leave
items which are not in a previous list and also take care of removing
duplicates.</p>
<p></p>
The challenge here is to do this without using vars and by only using
vals. If you use vars, the solution is trivial.</p>
<p></p>
The solution to this problem is using foldLeft. foldLeft essentially
gives you a way to operate on the elements of a list while aggregating
some state. The state you aggregate is the elements you've already seen
and the current result. The final solution looks like this:</p>
</p>
<p>
<script src="https://gist.github.com/1991619.js" type="text/javascript"><![CDATA[// <![CDATA[</p><p>// <![CDATA[</p><p>// <![CDATA[</p><p>// <![CDATA[</p><p> </p><p>// ]]]]]]]]]]><![CDATA[><![CDATA[><![CDATA[><![CDATA[></p><p>// ]]]]]]]]><![CDATA[><![CDATA[><![CDATA[></p><p>// ]]]]]]><![CDATA[><![CDATA[></p><p>// ]]]]><![CDATA[>]]></script>
</p>
<p></p>
And while we're at it, let's see how the C++ solution looks like, for
comparison. My C++ might be a bit rusty, but the result is longer.</p>
</p>
<p>
<script src="https://gist.github.com/1991674.js" type="text/javascript"><![CDATA[// <![CDATA[</p><p>// <![CDATA[</p><p>// <![CDATA[</p><p>// <![CDATA[</p><p> </p><p>// ]]]]]]]]]]><![CDATA[><![CDATA[><![CDATA[><![CDATA[></p><p>// ]]]]]]]]><![CDATA[><![CDATA[><![CDATA[></p><p>// ]]]]]]><![CDATA[><![CDATA[></p><p>// ]]]]><![CDATA[>]]></script>
</p>
</p>Simulating HTTP responses in Django.2011-11-11T19:15:00-08:002011-11-11T19:15:00-08:00Petko Minkovtag:pminkov.github.io,2011-11-11:/blog/simulating-http-responses-in-django.html<p>One of the most important aspects of your web application's client side
code is how it handles errors and slow connections. In the case of an
error, some reasonable error message should be displayed and in the case
of a slow connection, the user shouldn't be left staring at the …</p><p>One of the most important aspects of your web application's client side
code is how it handles errors and slow connections. In the case of an
error, some reasonable error message should be displayed and in the case
of a slow connection, the user shouldn't be left staring at the web app
wondering why did it freeze and you should at least display a spinning
wheel or some other "in progress" indicator.</p>
<p></p>
To solve that problem in Django, you can write a simple Django
application that simulates errors and slow connections. The way it works
is by installing a middleware component that catches HTTP requests and
returns error codes, or sleeps for some time before returning.</p>
<p></p>
I'll illustrate how to implement a very simple version of that idea.
Let's say that you'll have a handler sitting at "/http_simulate", which
when accessed will switch the server from error mode to healthy mode and
then back. The second thing you need is a middleware that knows what
mode the server is in and depending on that, fiddles with the HTTP
request and produces an error HTTP response.</p>
<p></p>
Here's how a very simple middleware class would look like:</p>
</p>
<p>
<script src="https://gist.github.com/1359978.js"> </script>
Notice how that middleware class ignores requests which go to the view
that switches to error mode. You want to be able to switch back to
normal mode once you switch to error mode.
</p>
<p>The second part you need is a view that switches the <code>IN_ERROR_MODE</code>
variable.</p>
</p>
<p>
<script src="https://gist.github.com/1359984.js"> </script>
After you have this code, you can just create a simple Django
application and install that middleware in your settings.py file. Be
careful to not have that turned on in your production environment.
</p>
<p>There are also many ways in which this can be extended:</p>
</p>
<ul>
<li>Support for different error codes.</li>
<li>Support for slow connections by sleeping before sending the
HTTP response.</li>
<li>Setting error mode for only specific urls, not for every request
sent to the server.</li>
</ul>
</p>