Jekyll2023-06-17T18:45:15+00:00https://lostmsu.github.io/feed.xmlVictor NovaA fight against the Big ChillThe Observable Universe, logarithmic scale2023-06-17T00:00:00+00:002023-06-17T00:00:00+00:00https://lostmsu.github.io/Visible-Universe<p><img src="https://habrastorage.org/getpro/habr/upload_files/f10/e21/585/f10e2158577d01935eed47d7007880a8.png" alt="The Observable Universe, logarithmic scale" /></p>Notes on setting up ReFS on Windows2022-07-09T00:00:00+00:002022-07-09T00:00:00+00:00https://lostmsu.github.io/ReFS<p>This is a runbook on setting up redundant storage on Windows.</p>
<h1 id="setting-up-storage-pool">Setting up storage pool</h1>
<p>You can create the storage pool from your drives using the ‘Manage Storage Spaces’
applet in the Windows Control Panel. The drives must be uninitialized.
To create a resilient space, you will need at least two drives in the pool,
and three or more drives for parity-based resilience. Note: you can always add more
drives to the pool later. Read a description of resilience types in this
<a href="https://docs.microsoft.com/en-us/azure-stack/hci/concepts/plan-volumes#with-four-or-more-servers">Microsoft article</a>
(it talks about servers, but the same applies to disks in local storage pools).</p>
<h1 id="creating-a-storage-space">Creating a storage space</h1>
<p>This will create an unformatted resilient (3 columns, e.g. 2 data + 1 parity)
80TB storage space, which does not require the whole capacity to be immediately available:</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">New-VirtualDisk</span><span class="w"> </span><span class="nt">-StoragePoolFriendlyName</span><span class="w"> </span><span class="s2">"YourPool"</span><span class="w"> </span><span class="nt">-ProvisioningType</span><span class="w"> </span><span class="nx">Thin</span><span class="w"> </span><span class="nt">-Interleave</span><span class="w"> </span><span class="nx">32KB</span><span class="w"> </span><span class="nt">-FriendlyName</span><span class="w"> </span><span class="nx">Resilient</span><span class="w"> </span><span class="nt">-Size</span><span class="w"> </span><span class="nx">80TB</span><span class="w"> </span><span class="nt">-ResiliencySettingName</span><span class="w"> </span><span class="nx">Parity</span><span class="w"> </span><span class="nt">-NumberOfColumns</span><span class="w"> </span><span class="nx">3</span><span class="w">
</span></code></pre></div></div>
<p><strong>NOTE</strong>: Pay attention to the <code class="language-plaintext highlighter-rouge">-Interleave</code> parameter. As described in <a href="https://storagespaceswarstories.com/storage-spaces-and-slow-parity-performance/">this article</a>,
it is important that when you format the disk in the next step, your file system uses
blocks whose size is evenly divisible by the interleave size.</p>
<p>This command will initialize the disk and assign it a drive letter:</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Get-VirtualDisk</span><span class="w"> </span><span class="nt">-FriendlyName</span><span class="w"> </span><span class="nx">Resilient</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Get-Disk</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Initialize-Disk</span><span class="w"> </span><span class="nt">-PassThru</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">New-Partition</span><span class="w"> </span><span class="nt">-UseMaximumSize</span><span class="w"> </span><span class="nt">-AssignDriveLetter</span><span class="w">
</span></code></pre></div></div>
<p>Now you can format it to ReFS:</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Format-Volume</span><span class="w"> </span><span class="nt">-DriveLetter</span><span class="w"> </span><span class="nx">XXX</span><span class="w"> </span><span class="nt">-FileSystem</span><span class="w"> </span><span class="nx">ReFS</span><span class="w"> </span><span class="nt">-AllocationUnitSize</span><span class="w"> </span><span class="p">(</span><span class="mi">32</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="n">YYY</span><span class="p">)</span><span class="w"> </span><span class="nt">-SetIntegrityStreams</span><span class="w"> </span><span class="nv">$True</span><span class="w">
</span></code></pre></div></div>
<h1 id="integrity-checking">Integrity checking</h1>
<p>You <strong>must</strong> enable periodic integrity checks on your ReFS volumes. Not doing so
means when you actually try accessing data many years later it might turn out to be
corrupted on all replicas, and will not be recoverable anymore. Periodic integrity
checks ensure all the replicas are correct, and if one replica is corrupted,
scanner will use the other replicas to repair the data. So the data could only be lost
if all replicas get corrupted between the integrity scans, which is much less likely.</p>
<p>To enable periodic checks, open Task Scheduler and find the scan tasks under
<code class="language-plaintext highlighter-rouge">Microsoft\Windows\Data Integrity Scan</code>. Then</p>
<ol>
<li>Pick the one with “Multiple triggers defined”</li>
<li>Adjust the trigger to the desired frequency/schedule</li>
<li><strong>MOST IMPORTANT</strong> Enable the trigger!</li>
</ol>
<p><img src="/images/Integrity-Scrub-Trigger.png" alt="Edit Trigger window showing the `Enabled` checkbox" /></p>
<p>See <a href="https://docs.microsoft.com/en-us/windows-server/storage/refs/integrity-streams#integrity-scrubber">Integrity Scrubber</a></p>
<p><strong>NOTE</strong>: There’s a disagreement about the meaning of ReFS integrity checking settings
and no guidance from Microsoft that I could find. File integrity must always be <em>Enabled</em>
for ReFS to be able to detect corruption (it goes to the Event Log), but <em>Enforced</em> state
which is <strong>enabled by default</strong> will confusingly make partially corrupted files extremely
hard to recover. So until the situation is resolved, I recommend setting <em>Enforced</em> to <code class="language-plaintext highlighter-rouge">False</code>.</p>
<h1 id="encryption">Encryption</h1>
<p>There’s a <a href="https://support.microsoft.com/en-us/topic/internal-sata-drives-show-up-as-removeable-media-1f806a64-8661-95a6-adc7-ce65a976c8dd">known issue</a>
with internal SATA drives on Windows that show up
as ‘Removable’ and could only be used with BitLocker To Go.
If you open BitLocker and your storage space is listed under ‘Removable’,
you’ll need to follow the steps mentioned later in
<a href="https://support.microsoft.com/en-us/topic/internal-sata-drives-show-up-as-removeable-media-1f806a64-8661-95a6-adc7-ce65a976c8dd#supArticleContent">the same article</a>
for each <em>physical</em> drive in your pool
(will likely need the <strong>Windows 8 and later</strong> option). A reboot might be required.</p>
<p>Once you’ve ensured, that your storage space is listed under ‘Fixed data drives’,
you can encrypt it.</p>This is a runbook on setting up redundant storage on Windows.Nevermind XOR - Deep Learning has an issue with Sin2022-04-18T00:00:00+00:002022-04-18T00:00:00+00:00https://lostmsu.github.io/Nevermind-XOR-Deep-Learning-has-an-isssue-with-Sin<p>More precisely, even the best neural networks can not be trained to approximate
periodic functions using stochastic gradient descent. (empirically, prove me wrong!)</p>
<h3>Contents</h3>
<ul id="markdown-toc">
<li><a href="#simulating-a-clock" id="markdown-toc-simulating-a-clock">Simulating a clock</a></li>
<li><a href="#simplifying-the-problem" id="markdown-toc-simplifying-the-problem">Simplifying the problem</a></li>
<li><a href="#why-cant-we-train-the-sin" id="markdown-toc-why-cant-we-train-the-sin">Why can’t we train the <code class="language-plaintext highlighter-rouge">sin</code>?</a></li>
<li><a href="#what-to-do-about-it" id="markdown-toc-what-to-do-about-it">What to do about it?</a></li>
</ul>
<h1 id="simulating-a-clock">Simulating a clock</h1>
<p>The problem that led me into this sinkhole is an attempt to model biological
rhythms. People sleep every 24 hours, the nature has a distinct 365 day cycle,
and over a month the Moon goes from new to full and back. I wanted to capture
that repetition relative to the current instant in time, which led
to the Clock Problem:</p>
<p>Given the number of seconds since some moment T0 defined to be 0 days 0:00:00,
train a neural network to approximate the numbers you would see on a digital
clock. For example, <code class="language-plaintext highlighter-rouge">-1</code> (second) would be <code class="language-plaintext highlighter-rouge">23:59:59</code>.</p>
<p>Expecting that to be a dead simple task, I built an infinite dataset, that
would sample a random instant in time from a period of 120 years, and fed it
into a <a href="https://www.vincentsitzmann.com/siren/">SIREN</a> - neural network with
sinusoidal activations.</p>
<p>To my surprise, despite playing with its frequency scale hyperparameter,
the network, essentially, never converged.</p>
<p>I tried to use regular MLP with GELU activations, and got approximately
the same result.</p>
<p>Research on the topic only brought <a href="https://proceedings.neurips.cc/paper/2020/file/1160453108d3e537255e9f7b931f4e90-Paper.pdf">Neural Networks Fail to Learn Periodic Functions and How to Fix It</a>,
which, as you might have guessed it, did not really work on the Clock Problem.
Their <code class="language-plaintext highlighter-rouge">x + sin(x) ** 2</code> activation only really worked when the number of full cycles
in the dataset was less than the number of paramters of the network, which
completely misses the point.</p>
<h1 id="simplifying-the-problem">Simplifying the problem</h1>
<p>You can quickly see how inappropriate gradient descent is for the problem if we
just simplify it a little. Let’s try approximating this trivial function:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">FREQUENCY_SCALE</span> <span class="o">=</span> <span class="mi">31</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
<span class="k">return</span> <span class="n">torch</span><span class="p">.</span><span class="n">sin</span><span class="p">(</span><span class="n">x</span> <span class="o">*</span> <span class="n">FREQUENCY_SCALE</span><span class="p">)</span>
</code></pre></div></div>
<p>There is a a PyTorch module, that surely should solve the problem:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Sin</span><span class="p">(</span><span class="n">torch</span><span class="p">.</span><span class="n">nn</span><span class="p">.</span><span class="n">Module</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">super</span><span class="p">().</span><span class="n">__init__</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">freq</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">nn</span><span class="p">.</span><span class="n">Parameter</span><span class="p">(</span><span class="n">torch</span><span class="p">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
<span class="k">return</span> <span class="n">torch</span><span class="p">.</span><span class="n">sin</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">freq</span> <span class="o">*</span> <span class="n">x</span><span class="p">)</span>
</code></pre></div></div>
<p>Here, we only need to find the frequency, and the module will match our target
function exactly! Let’s try it out:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">net</span> <span class="o">=</span> <span class="n">Sin</span><span class="p">()</span>
<span class="n">opt</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">optim</span><span class="p">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">net</span><span class="p">.</span><span class="n">parameters</span><span class="p">(),</span> <span class="n">lr</span><span class="o">=</span><span class="mf">0.0001</span><span class="p">)</span>
<span class="n">BATCH_SIZE</span> <span class="o">=</span> <span class="mi">32</span>
<span class="k">for</span> <span class="n">batch_idx</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1000</span><span class="o">*</span><span class="mi">1000</span><span class="p">):</span>
<span class="n">opt</span><span class="p">.</span><span class="n">zero_grad</span><span class="p">()</span>
<span class="n">batch</span> <span class="o">=</span> <span class="p">(</span><span class="n">torch</span><span class="p">.</span><span class="n">rand</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="p">[</span><span class="n">BATCH_SIZE</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span> <span class="o">*</span> <span class="mi">2</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="mi">1000</span>
<span class="n">out</span> <span class="o">=</span> <span class="n">net</span><span class="p">(</span><span class="n">batch</span><span class="p">)</span>
<span class="n">expected</span> <span class="o">=</span> <span class="n">func</span><span class="p">(</span><span class="n">batch</span><span class="p">)</span>
<span class="n">loss</span> <span class="o">=</span> <span class="p">((</span><span class="n">out</span> <span class="o">-</span> <span class="n">expected</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">).</span><span class="n">mean</span><span class="p">()</span>
<span class="n">loss</span><span class="p">.</span><span class="n">backward</span><span class="p">()</span>
<span class="n">opt</span><span class="p">.</span><span class="n">step</span><span class="p">()</span>
<span class="k">if</span> <span class="n">batch_idx</span> <span class="o">%</span> <span class="mi">1000</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="n">loss</span><span class="p">.</span><span class="n">detach</span><span class="p">().</span><span class="n">cpu</span><span class="p">().</span><span class="n">item</span><span class="p">()</span><span class="si">}</span><span class="s">'</span><span class="p">)</span>
</code></pre></div></div>
<p>If you run this on your machine, you will see something like this:</p>
<p>0.775499165058136
1.3729740381240845
1.0878400802612305
0.7583212852478027
1.3061308860778809
0.6976296305656433
1.0671122074127197
0.9739978909492493
0.947789192199707</p>
<p>The loss just floats around 1 and never converges.</p>
<p>But we actually know the answer! Just insert this line:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">net</span><span class="p">.</span><span class="n">freq</span><span class="p">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">tensor</span><span class="p">([</span><span class="mi">31</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">torch</span><span class="p">.</span><span class="n">float32</span><span class="p">)</span>
</code></pre></div></div>
<p>and your loss will turn to zero instantly:</p>
<p>0.0
0.0
0.0
0.0</p>
<h1 id="why-cant-we-train-the-sin">Why can’t we train the <code class="language-plaintext highlighter-rouge">sin</code>?</h1>
<p>For the given <code class="language-plaintext highlighter-rouge">x0</code> the derivative of our scaled <code class="language-plaintext highlighter-rouge">sin</code> with respect to <code class="language-plaintext highlighter-rouge">freq</code>
parameter is <code class="language-plaintext highlighter-rouge">x0 * cos(x0 * freq)</code>. There are two things to note:</p>
<ol>
<li>The scale of the derivative value depends on how far x0 is from 0. This
is bad, as really that dependency only makes sense within
<code class="language-plaintext highlighter-rouge">[-x0/freq*pi, +x0/freq*pi]</code> interval.</li>
<li>The derivative does not really point where we want to go. Instead, its
direction only depends on whether the <code class="language-plaintext highlighter-rouge">x0 * freq</code> is closer to <code class="language-plaintext highlighter-rouge">2n*pi</code>
or <code class="language-plaintext highlighter-rouge">(2n+1)*pi</code> for some <code class="language-plaintext highlighter-rouge">n</code>. And that value will vary wildly for different
samples in the batch.</li>
</ol>
<p>This is how the gradient of the <code class="language-plaintext highlighter-rouge">freq</code> paramter looks like on a large random
batch of points:</p>
<p><img src="/images/Sin-Grad.png" alt="freq parameter gradient is all over the place" /></p>
<p>Can you spot the 0 at 31? As you can see, around it the gradient is all over
the place.</p>
<p>Even in the immediate vicinity of 31, it does not behave well:</p>
<p><img src="/images/Sin-Close-Grad.png" alt="freq parameter gradient near 31 is also all over the place" /></p>
<h1 id="what-to-do-about-it">What to do about it?</h1>
<p>I don’t know. Perhaps a form of Fourier transform or a wavelet transform
would help. Time to beef up on signal processing.</p>More precisely, even the best neural networks can not be trained to approximate periodic functions using stochastic gradient descent. (empirically, prove me wrong!) Contents Simulating a clock Simplifying the problem Why can’t we train the sin? What to do about it? Simulating a clock The problem that led me into this sinkhole is an attempt to model biological rhythms. People sleep every 24 hours, the nature has a distinct 365 day cycle, and over a month the Moon goes from new to full and back. I wanted to capture that repetition relative to the current instant in time, which led to the Clock Problem: Given the number of seconds since some moment T0 defined to be 0 days 0:00:00, train a neural network to approximate the numbers you would see on a digital clock. For example, -1 (second) would be 23:59:59. Expecting that to be a dead simple task, I built an infinite dataset, that would sample a random instant in time from a period of 120 years, and fed it into a SIREN - neural network with sinusoidal activations. To my surprise, despite playing with its frequency scale hyperparameter, the network, essentially, never converged. I tried to use regular MLP with GELU activations, and got approximately the same result. Research on the topic only brought Neural Networks Fail to Learn Periodic Functions and How to Fix It, which, as you might have guessed it, did not really work on the Clock Problem. Their x + sin(x) ** 2 activation only really worked when the number of full cycles in the dataset was less than the number of paramters of the network, which completely misses the point. Simplifying the problem You can quickly see how inappropriate gradient descent is for the problem if we just simplify it a little. Let’s try approximating this trivial function: FREQUENCY_SCALE = 31 def func(x): return torch.sin(x * FREQUENCY_SCALE) There is a a PyTorch module, that surely should solve the problem: class Sin(torch.nn.Module): def __init__(self): super().__init__() self.freq = torch.nn.Parameter(torch.randn(1)) def forward(self, x): return torch.sin(self.freq * x) Here, we only need to find the frequency, and the module will match our target function exactly! Let’s try it out: net = Sin() opt = torch.optim.SGD(net.parameters(), lr=0.0001) BATCH_SIZE = 32 for batch_idx in range(1000*1000): opt.zero_grad() batch = (torch.rand(size=[BATCH_SIZE, 1], device=device) * 2 - 1) * 1000 out = net(batch) expected = func(batch) loss = ((out - expected) ** 2).mean() loss.backward() opt.step() if batch_idx % 1000 == 0: print(f'{loss.detach().cpu().item()}') If you run this on your machine, you will see something like this: 0.775499165058136 1.3729740381240845 1.0878400802612305 0.7583212852478027 1.3061308860778809 0.6976296305656433 1.0671122074127197 0.9739978909492493 0.947789192199707 The loss just floats around 1 and never converges. But we actually know the answer! Just insert this line: net.freq.data = torch.tensor([31], dtype=torch.float32) and your loss will turn to zero instantly: 0.0 0.0 0.0 0.0 Why can’t we train the sin? For the given x0 the derivative of our scaled sin with respect to freq parameter is x0 * cos(x0 * freq). There are two things to note: The scale of the derivative value depends on how far x0 is from 0. This is bad, as really that dependency only makes sense within [-x0/freq*pi, +x0/freq*pi] interval. The derivative does not really point where we want to go. Instead, its direction only depends on whether the x0 * freq is closer to 2n*pi or (2n+1)*pi for some n. And that value will vary wildly for different samples in the batch. This is how the gradient of the freq paramter looks like on a large random batch of points: Can you spot the 0 at 31? As you can see, around it the gradient is all over the place. Even in the immediate vicinity of 31, it does not behave well: What to do about it? I don’t know. Perhaps a form of Fourier transform or a wavelet transform would help. Time to beef up on signal processing.Use Microsoft Store to trick unsuspecting users to give you all their data2020-02-03T00:00:00+00:002020-02-03T00:00:00+00:00https://lostmsu.github.io/Steal-Data-Using-Microsoft-Store<p>Privacy is hard these days. Tech companies want to know everything about you to
show more and more advertisement, or to sell the data to the third parties, who
then resell to the next ones, then further, until a a toy vendor knows more
about your kids, than you do.</p>
<p><img src="https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fmedia.makeameme.org%2Fcreated%2FYour-data-Give.jpg&f=1&nofb=1" alt="Your Data: Give it to us meme with Lt. Commander Data" /></p>
<p>The app permission systems, that became popular with Android and iOS are a
godsend. Now your bookreader app does not know your location, who your friends
are, and can’t upload your private photos to their servers as a part of
“analytics telemetry”.</p>
<p>Microsoft introduced a similar system with Windows 8 for the apps installed
from the Microsoft Store, and expanded it in Windows 10 to any app packaged
in a special way. So when you install an app from Store, you can be reasonably
sure, that it does only what it is supposed to, right? Not so fast!</p>
<!--more-->
<h1 id="dont-blame-the-messenger">Don’t blame the Messenger</h1>
<p>Facebook Messenger is famous for trying to obtain data. It used to insist on
uploading your entire contact list to Facebook. Same goes for call and sms
history. Text anyone in your phone? Here, can you spot the fine print:</p>
<p><img src="https://about.fb.com/wp-content/uploads/2018/03/opt-in_screen-1.png?resize=576%2C1024" alt="Text anyone in your phone? Turn on!" /></p>
<p>I am sure it is used only for your convenience (khe-khe, viral marketing, khe-khe).</p>
<p>Fortunately, even if somehow this option got overlooked, and you installed the
app from Microsoft Store in 2018, you could ensure it does not have access to
your data.</p>
<p><img src="/images/ContactsAccess.png" alt="Contacts permission for Messenger in Windows Setting" /></p>
<p>Similar permissions exist for call history, pictures, location, etc</p>
<p>No viral marketing or personalized ads for you, Mark!</p>
<p><img src="https://fossbytes.com/wp-content/uploads/2016/02/mark-zuckerberg-sad-shuts-down-free-basics.jpg" alt="Mark Zuckerberg is sad" /></p>
<h1 id="microsoft-to-the-rescue-with-automatic-updates">Microsoft to the rescue with automatic updates</h1>
<p>That was all great until about 8 months ago. That is when I noticed, that
one of the apps installed from the Store specifically for not asking
any permissions whatsoever started showing a tray icon.</p>
<p>Tray icons are “classic” Windows APIs, that are not yet available to properly
sandboxed Store apps. To show one the app had to obtain unrestricted permissions
on your machine, same as the apps installed from a 3rd party.</p>
<p><img src="https://external-content.duckduckgo.com/iu/?u=http%3A%2F%2F3.bp.blogspot.com%2F-Pv29dGQwIMI%2FUFcMD6hgcyI%2FAAAAAAAAFHI%2Fg_z4acYUqT8%2Fs1600%2FttUntitled-2.jpg&f=1&nofb=1" alt="Windows XP Tray Icon" /></p>
<p>So I went to the app’s page on Microsoft Store, and saw, that the new version has
this permission. Which would be fine, if the Store would ask me if I want to give
that app unrestricted permissions before updating it. But it did not.</p>
<p>Back then I reported the problem to Microsoft, thinking that they just probably
overlooked it, and it is a bug, that will soon be fixed.</p>
<h1 id="8-months-later-microsoft-still-does-not-care-enough">8 months later Microsoft still does not care enough</h1>
<p>Coming back to Messenger: that app is notoriously buggy and did not work
on my machine for a while, forcing to use the web version. I’ve been
periodically launching it to see if Facebook fixed it, and today it finally
started with a new login dialog. Which, of course, immediately made
my inner paranoid very suspicious.</p>
<p>So I opened Messenger’s page on Windows Store, and saw this:
<img src="/images/AccessAll.png" alt="Facebook Messenger has access to all your data" /></p>
<p>Looks like Windows Store silently updated it, expanding the permissions
to give Facebook access to all my data without consulting with me first.
Meaning the issue I reported nearly a year ago is still not fixed.</p>
<p>So, to conclude</p>
<h1 id="5-steps-you-need-to-take-to-own-users-machine">5 steps you need to take to own user’s machine</h1>
<ol>
<li>Create a sandboxed app, and publish it to Microsoft Store. Don’t ask for
any permissions (maybe a few). Privacy-conscious users will love it!</li>
<li>Find an excuse to get Microsoft approve unrestricted access for your app.
Tray icon might do, or someting equally innocent.</li>
<li>After 2 is approved, add code siphoning data out/taking control when you need
it (formal update submission process rarely checks for it), but keep the core
functionality.</li>
<li>…</li>
<li>PROFIT</li>
</ol>
<p>All users, that installed your app before step 2 thinking it is 100% safe for them to use will be automatically updated to the latest version, and Store will give
it a green ticket to user’s data.</p>
<p>P.S. The sad irony is I though it was a good idea to distribute a cryptocurrency
mining app through the Store, as it would be a safe source for users. Until
Microsoft <a href="2020-01-15-Microsoft-bans-cryptocurrency-miners-from-Store">banned them for “security”</a>.</p>Privacy is hard these days. Tech companies want to know everything about you to show more and more advertisement, or to sell the data to the third parties, who then resell to the next ones, then further, until a a toy vendor knows more about your kids, than you do. The app permission systems, that became popular with Android and iOS are a godsend. Now your bookreader app does not know your location, who your friends are, and can’t upload your private photos to their servers as a part of “analytics telemetry”. Microsoft introduced a similar system with Windows 8 for the apps installed from the Microsoft Store, and expanded it in Windows 10 to any app packaged in a special way. So when you install an app from Store, you can be reasonably sure, that it does only what it is supposed to, right? Not so fast!Microsoft bans cryptocurrency miners from Store2020-01-15T00:00:00+00:002020-01-15T00:00:00+00:00https://lostmsu.github.io/Microsoft-bans-cryptocurrency-miners-from-Store<p>Today I was notified, that it is no longer possible to submit updates
to my user-friendly Ethereum miner to Windows Store due to
the new Microsoft policies.</p>
<p>What a shame.</p>
<p>UPD. You can now reinstall miner from <a href="Microsoft-bans-cryptocurrency-miners-from-Store/#Update-Miner-is-available-on-Lost-Tech-website">Lost Tech Downloads</a></p>
<!--more-->
<p>The whole idea of distributing it through the Store was for users to
have a safe place to discover and download a cryptocurrency miner,
as opposed to looking for one on random Internet forums with real risk to
get malware in the package. And yet Microsoft banned miners from their store for
<a href="https://docs.microsoft.com/en-us/windows/uwp/publish/store-policies#102-security">“security”</a>.
Ironically, that’s the same policy section, that bans 3rd-party browser engines.</p>
<p><img src="https://sophosnews.files.wordpress.com/2011/08/nocrypto-square.png" alt="No Crypto sign" /></p>
<p>By banning miners, Microsoft is probably trying to prevent employees in companies
worldwide from using corporate hardware for personal profit. But by doing so
it also hurts consumers, who now have to take the above mentioned risks on themselves
if they want to adopt this tech at home (my wife uses it as a “free” room heater).
They could certainly just have a special mark on the app, and have a group policy
denying the installation, that enterprises could enable, rather than denying
the access to everyone.</p>
<p>Now I’ll have to spent a few days developing an alternative distribution system,
and probably will loose a significant chunk of existing users.
IMHO, that policy will hurt overall tech adoption too.</p>
<h2 id="update-miner-is-available-from-lost-tech-website">UPDATE: Miner is available from Lost Tech website</h2>
<p>Download from <a href="https://losttech.software/Downloads/Mine/">Lost Tech Downloads</a></p>
<p>If you had one installed from Store, backup</p>
<p><code class="language-plaintext highlighter-rouge">C:\Users\<YOUR USER>\AppData\Local\Packages\LostTechLLC.MineETH_kdyhxf5sz30e2\LocalState\Mine.config</code></p>
<p>Then uninstall the Store one, and install the new one.</p>
<p>Copy your config backup and overwrite the new config location:</p>
<p><code class="language-plaintext highlighter-rouge">AppData\Local\Packages\LostTechLLC.MineETH_aptdcfzknqkda\LocalState\Mine.config</code></p>Today I was notified, that it is no longer possible to submit updates to my user-friendly Ethereum miner to Windows Store due to the new Microsoft policies. What a shame. UPD. You can now reinstall miner from Lost Tech DownloadsChromium is no longer supported on Chocolatey2019-10-25T00:00:00+00:002019-10-25T00:00:00+00:00https://lostmsu.github.io/Chromium<p>Had to switch to Brave today, as Chromium has not been getting new versions on Chocolatey,
which is a security issue. It is stuck at v75 while v76, v77, and v78 have been out for a while.</p>Had to switch to Brave today, as Chromium has not been getting new versions on Chocolatey, which is a security issue. It is stuck at v75 while v76, v77, and v78 have been out for a while.Git Explode2019-10-24T00:00:00+00:002019-10-24T00:00:00+00:00https://lostmsu.github.io/Git-Explode<p>A story of lost and recovered code for a project I just created.</p>
<!--more-->
<h2 id="boom">Boom!</h2>
<p>While upgrading <a href="https://losttech.software/gradient.html">Gradient</a> to TensorFlow 1.15,
I faced the need for a way to create a
<a href="https://en.wikipedia.org/wiki/Symbolic_link">symbolic directory link</a> with C#.
Quick search on <a href="https://nuget.org">NuGet</a> only gave 3 packages, none of which were
cross-platform to a complete disappointment. So I decided to make my own.</p>
<p>Quick 2 hours later, I’ve got a local Visual Studio solution with implementations
for Windows and Unix-like OSes. It was time to publish it on GitHub. So I did two things:</p>
<ol>
<li>Created a new repository on GitHub itself, with a README and LICENSE files</li>
<li>In Visual Studio, hit “Add to Source Control” and selected Git</li>
</ol>
<p>Now I needed to merge the two, so I hit publish in Visual Studio, and pointed it to GitHub.
That set the <code class="language-plaintext highlighter-rouge">origin</code> remote to my GitHub repo, but did not publish local changes,
as the local <code class="language-plaintext highlighter-rouge">master</code> branch and the remote one did not have a common ancestor.</p>
<p>“OK”, I though, “let’s fetch, make local <code class="language-plaintext highlighter-rouge">master</code> point to <code class="language-plaintext highlighter-rouge">origin/master</code>, from GitHub
then commit local files”. So I did</p>
<pre><code class="language-PowerShell">git fetch
git reset --hard origin/master
</code></pre>
<p><img src="/images/Explosion.jpg" alt="Explosion" /></p>
<p>Back in Visual Studio all the source code is gone! So are the project files!</p>
<pre><code class="language-PowerShell">> git status
HEAD detached from c988afd
nothing to commit, working tree clean
</code></pre>
<h2 id="denial">Denial</h2>
<p>If you ever did <code class="language-plaintext highlighter-rouge">git reset</code> without thinking before and recovered, you know a “magical”
command <code class="language-plaintext highlighter-rouge">git reflog</code>, which shows recent commits you worked with. To be precise it
shows which commits your repository was ever checked out at, so if you did
<code class="language-plaintext highlighter-rouge">git checkout AAA</code> then <code class="language-plaintext highlighter-rouge">git checkout BBB</code> then <code class="language-plaintext highlighter-rouge">git reset --hard CCC</code> it will show <code class="language-plaintext highlighter-rouge">AAA;BBB;CCC</code>,
and you can return to any of them using <code class="language-plaintext highlighter-rouge">git checkout AAA</code>, as long as you never had
git <a href="https://git-scm.com/docs/git-gc">collect garbage</a>.</p>
<p>For example:</p>
<pre><code class="language-git">> git reflog
...
c988afd HEAD@{4}: checkout: moving from master to c988afd
ea279fb HEAD@{5}: reset: moving to origin/master
c988afd HEAD@{6}: commit (initial): Add .gitignore and .gitattributes.
</code></pre>
<p>Unfortunately, <code class="language-plaintext highlighter-rouge">git checkout c988afd</code> did not restore the files, as I expected.</p>
<p><img src="/images/Sad.jpg" alt="Sad man" /></p>
<h2 id="anger">Anger</h2>
<p><strong>But why?</strong> The message for the initial commit gave a hint: I never checked in those files
before <code class="language-plaintext highlighter-rouge">git reset</code>, but they were staged. And, turns out, when you switch to a commit tree
with a different initial commit, <code class="language-plaintext highlighter-rouge">git reset</code> completely drops all the staged changes!</p>
<p>And Windows integrated backup - File History - by that time did not have a chance to make copies!</p>
<h2 id="bargaining">Bargaining</h2>
<p>A quick search gave back
<a href="https://stackoverflow.com/questions/5788037/recover-from-git-reset-hard">Recover from git reset –hard</a>
on StackOverflow.</p>
<p>So I did <code class="language-plaintext highlighter-rouge">git fsck --lost-found</code>, which created a folder <code class="language-plaintext highlighter-rouge">.git\lost-found\other</code>
with a single file in it, containing just its own name: <code class="language-plaintext highlighter-rouge">97cc6b041928db3f3d282bbd30f5d1e276e47b19</code>.</p>
<p>How that would help was not immediately clear:</p>
<pre><code class="language-PowerShell">> git show 97cc6b041928db3f3d282bbd30f5d1e276e47b19
tree 97cc6b041928db3f3d282bbd30f5d1e276e47b19
.gitattributes
.gitignore
IO.Links.sln
play/
src/
> git show 97cc6b041928db3f3d282bbd30f5d1e276e47b19 -- IO.Links.sln
tree 97cc6b041928db3f3d282bbd30f5d1e276e47b19
.gitattributes
.gitignore
IO.Links.sln
play/
src/
❯ git checkout 97cc6b041928db3f3d282bbd30f5d1e276e47b19
fatal: Cannot switch branch to a non-commit '97cc6b041928db3f3d282bbd30f5d1e276e47b19'
</code></pre>
<p><strong>What is a git tree?</strong> I did not know.</p>
<h2 id="depression">Depression</h2>
<p>The first search pointed to
<a href="https://www.git-scm.com/docs/git-worktree">Git worktree</a>, but</p>
<pre><code class="language-PowerShell">❯ git worktree move 97cc6b041928db3f3d282bbd30f5d1e276e47b19 "..\recover"
fatal: '97cc6b041928db3f3d282bbd30f5d1e276e47b19' is not a working tree
</code></pre>
<p>So a <em>tree</em> is not a <em>working tree</em>, OK.</p>
<p><strong>Sigh.</strong></p>
<h2 id="acceptable">Acceptable</h2>
<p>Now searching <code class="language-plaintext highlighter-rouge">git restore tree -"working"</code>…</p>
<p><a href="https://git-scm.com/docs/git-ls-tree">Bingo!</a> (ls-tree)</p>
<pre><code class="language-PowerShell">❯ git ls-tree 97cc6b041928db3f3d282bbd30f5d1e276e47b19
100644 blob 1ff0c423042b46cb1d617b81efb715defbe8054d .gitattributes
100644 blob 4ce6fddec962ff3b86038d9939b6be5dfc1e6351 .gitignore
100644 blob c77dfcf2c002aae9fefa37273e9a82e290324ed1 IO.Links.sln
040000 tree 7d34ff63a2a130c9e8b7946e535ce83755db1ac7 play
040000 tree 91aa24f2b82c94d1ec4943b3747cd4d46a14db21 src
</code></pre>
<p>So there are blobs!</p>
<p><img src="/images/EndOfTunnel.jpg" alt="Light in at the tunnel end" /></p>
<p>But no more <code class="language-plaintext highlighter-rouge">*tree*</code> commands.</p>
<p>More searching –> <a href="https://git-scm.com/docs/git-restore">Git Restore</a></p>
<pre><code class="language-PowerShell">> git restore -s 97cc6b041928db3f3d282bbd30f5d1e276e47b19 -S -W
git: 'restore' is not a git command. See 'git --help'.
The most similar command is
remote
</code></pre>
<p><strong>o_O</strong> but it is right there, in docs!</p>
<pre><code class="language-PowerShell">> git --version
git version 2.17.1
</code></pre>
<p>Ah. Docs are for version <em>2.23.0</em></p>
<pre><code class="language-PowerShell">choco install git
git --version
git version 2.23.0.windows.1
</code></pre>
<p>Finally:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> git restore -s 97cc6b041928db3f3d282bbd30f5d1e276e47b19 -S -W -- .
> git status
HEAD detached at c988afd
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
new file: IO.Links.sln
new file: play/Program.cs
new file: play/play.csproj
new file: src/IFileSystemLinks.cs
new file: src/Links.csproj
new file: src/Symlink.cs
new file: src/UnixLinks.cs
new file: src/WindowsLinks.cs
</code></pre></div></div>
<iframe src="https://giphy.com/embed/VbrGKu56dceVa" width="480" height="270" frameborder="0" class="giphy-embed" allowfullscreen=""></iframe>
<h2 id="conclusion">Conclusion</h2>
<p><strong>1AM</strong></p>
<pre><code class="language-PowerShell">> git fsck --lost-found
> ls .\.git\lost-found\other
> git restore -s HASH_FROM_OTHER -S -W -- .
</code></pre>
<p>Project is restored and out on GitHub and NuGet: <a href="https://github.com/losttech/IO.Links">IO.Links</a>
- cross-platform symlink and hardlink management for .NET.
(really: untested symlink creation only v0.0.1 at the time of the post).</p>
<h3 id="update-2019-10-26">Update 2019-10-26</h3>
<p>As Ape3000 mentioned in the <a href="https://www.reddit.com/r/git/comments/dn3i2z/successfully_recovered_after_git_reset_hard_with/f598bed/">Reddit thread</a>, updating git is not necessary.
<code class="language-plaintext highlighter-rouge">git restore</code> can be replaced with a variant of checkout:</p>
<pre><code class="language-PowerShell"># git checkout (-p|--patch) [<tree-ish>] [--] [<paths>…]
> git checkout HASH_FROM_OTHER -- .
</code></pre>A story of lost and recovered code for a project I just created.Deep learning - Not C#2019-10-15T00:00:00+00:002019-10-15T00:00:00+00:00https://lostmsu.github.io/Not-CSharp<p>In Silicon Valley season 4, Jian-Yang builds an AI app that identifies pictures of hotdogs.
Today I am going to make a similar app to identify C# from code fragments (in case you forget
how C# looks like!).</p>
<p>Look at this Python trying to pretend to be your favorite language:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">var</span> <span class="o">=</span> <span class="k">await</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
<span class="n">switch</span><span class="p">(</span><span class="n">hello</span><span class="p">)</span>
</code></pre></div></div>
<p>Of course the whole thing from building and training a deep convolutional network
to the cross-platform UI will be written in .NET + TensorFlow.</p>
<p><img src="/images/NotCSharp.png" alt="Not C# screenshot, showing Python detected" /></p>
<p><em>Advanced-level tutorial on training a deep convnet to classify language from code fragments</em></p>
<!--more-->
<h3>Contents</h3>
<ul id="markdown-toc">
<li><a href="#building--training-deep-neural-network" id="markdown-toc-building--training-deep-neural-network">Building & Training Deep Neural Network</a> <ul>
<li><a href="#64-by-64-blocks-of-text" id="markdown-toc-64-by-64-blocks-of-text">64 by 64 blocks of text</a></li>
<li><a href="#preparing-data" id="markdown-toc-preparing-data">Preparing data</a></li>
<li><a href="#building-a-convolutional-network" id="markdown-toc-building-a-convolutional-network">Building a convolutional network</a></li>
<li><a href="#training-the-network" id="markdown-toc-training-the-network">Training the network</a></li>
<li><a href="#training-issues" id="markdown-toc-training-issues">Training Issues</a></li>
<li><a href="#hooray" id="markdown-toc-hooray">Hooray!</a></li>
</ul>
</li>
<li><a href="#making-an-app-avalonia-ui" id="markdown-toc-making-an-app-avalonia-ui">Making an app: Avalonia UI</a></li>
<li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li>
</ul>
<h1 id="building--training-deep-neural-network">Building & Training Deep Neural Network</h1>
<h2 id="64-by-64-blocks-of-text">64 by 64 blocks of text</h2>
<p>Normally to process a sequence of any kind (in this case - characters), one would use an <a href="https://en.wikipedia.org/wiki/LSTM">LSTM</a> - long-short term memory network. However, getting an LSTM network
to recognize a programming language of an entire file is too easy and boring.</p>
<p>Instead, Not C# AI will take as input a 64 by 64 block of characters from a text file, starting at
an arbitrary position.</p>
<p>For example, this file</p>
<div class="language-cpp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Accelerate</span><span class="o">*</span> <span class="n">AccelerateFactory</span><span class="o">::</span><span class="n">Create</span><span class="p">(</span>
<span class="kt">int</span> <span class="n">sample_rate_hz</span><span class="p">,</span>
<span class="kt">size_t</span> <span class="n">num_channels</span><span class="p">,</span>
</code></pre></div></div>
<p>from position (2,5) will look like this:</p>
<div class="language-cpp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">nt</span> <span class="n">sample_rate_hz</span><span class="p">,</span>
<span class="n">ize_t</span> <span class="n">num_channels</span><span class="p">,</span>
</code></pre></div></div>
<p>If a file/line ends too soon, the rest will be padded with space characters to make a full 64 by 64 block.</p>
<p>Every character will be converted to 0-255 range, and the whole input will look like
a 64x64 grayscale image. You can see an example in the center of the pic below.
Text on the right helps to understand which part of the code it represents,
when you move the starting point.</p>
<p><img src="/images/NotCSharp-Parts.png" alt="Not C# screenshot, showing the image, that network sees" /></p>
<p>Now try to guess the language from this image :)</p>
<p><img src="/images/CSharp-NOT-Code-IMG.png" alt="Code, encoded as a grayscale image" /></p>
<p><em>HINT: there’s a single character in 2nd line from the last</em></p>
<h2 id="preparing-data">Preparing data</h2>
<p>Turned out my <code class="language-plaintext highlighter-rouge">Projects</code> folder has plenty of code with the following extensions: <code class="language-plaintext highlighter-rouge">.cs</code>, <code class="language-plaintext highlighter-rouge">.py</code>,
<code class="language-plaintext highlighter-rouge">.h</code>, <code class="language-plaintext highlighter-rouge">.cc</code>, <code class="language-plaintext highlighter-rouge">.c</code>, <code class="language-plaintext highlighter-rouge">.tcl</code>, <code class="language-plaintext highlighter-rouge">.java</code>, <code class="language-plaintext highlighter-rouge">.sh</code>, many coming from various open source libraries.
I also downloaded ~5 top trending GitHub repositories to have
some variety in the code styles.</p>
<p>All files needed some preprocessing on load: replaced tab character with spaces, and any characters with code
above 255 with underscores (so that each character took exactly 1 byte, <em>sorry, Unicode</em>).
Also replaced all other non-newline whitespace characters with code 255 to make them stand out.</p>
<p>I set aside 10% files of each type for validation. They are used at the end of the training
to ensure the model does not just learn patterns specific to the files I train it on.</p>
<p>There were several thousand of files of each type, more of some, less of others. To ensure the model will
not be biased towards more popular languages, for each file type the training code samples ~50,000 64x64 blocks,
no matter how many files of this type there are. The process is simple: first, pick a random file with
specific extension, then pick a random line and column number to start sample from, then copy a 64x64 block of
characters from that position into a <code class="language-plaintext highlighter-rouge">byte</code> array, containing training samples.
Picking a file first ensures large files will not be overrepresented. The process is repeated
50,000 times for each extension to obtain enough samples (<a href="https://github.com/losttech/Gradient-Samples/blob/1495521b6b8ba1b18c1167b838b08372bca8a4f3/CSharpOrNot/TrainCommand.cs#L130">sampling code</a>).</p>
<p>So inputs are 64x64 byte arrays, where character code simply turns into brightness of the “pixel”.
In the end the network will only see the image in the center as input. Text on the right helps to
understand which part of code it represents. Try to guess the language from that image, ha!</p>
<h2 id="building-a-convolutional-network">Building a convolutional network</h2>
<p>2D block picture-like representation suggests to use a <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network">convolutional network</a> architecture.
One of the recent ones is called <a href="https://towardsdatascience.com/an-overview-of-resnet-and-its-variants-5281e2f56035">ResNet</a>.
Not C# will use a simplified version.</p>
<p>ResNet consists of blocks, which stack 3 <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D">convolutional layers</a>,
connected by <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/BatchNormalization">batch normalization</a>
with a variant of <a href="https://mlfromscratch.com/activation-functions-explained/">ReLU activation</a>.
The output of the stack is concatenated with the input before the last activation, which is called
skip-connection. Skip-connection is there to help errors propagate to the blocks, that are closer
to the image input, than to the classifier output. The block layers are composed in the <code class="language-plaintext highlighter-rouge">CallImpl</code>
function.</p>
<p>The ResNet block code is adopted from the official TensorFlow tutorial for <a href="https://www.tensorflow.org/tutorials/customization/custom_layers#models_composing_layers">composing layers</a>.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">ResNetBlock</span><span class="p">:</span> <span class="n">Model</span> <span class="p">{</span>
<span class="k">const</span> <span class="kt">int</span> <span class="n">PartCount</span> <span class="p">=</span> <span class="m">3</span><span class="p">;</span>
<span class="k">readonly</span> <span class="n">PythonList</span><span class="p"><</span><span class="n">Conv2D</span><span class="p">></span> <span class="n">convs</span> <span class="p">=</span> <span class="k">new</span> <span class="n">PythonList</span><span class="p"><</span><span class="n">Conv2D</span><span class="p">>();</span>
<span class="k">readonly</span> <span class="n">PythonList</span><span class="p"><</span><span class="n">BatchNormalization</span><span class="p">></span> <span class="n">batchNorms</span> <span class="p">=</span> <span class="k">new</span> <span class="n">PythonList</span><span class="p"><</span><span class="n">BatchNormalization</span><span class="p">>();</span>
<span class="k">readonly</span> <span class="n">PythonFunctionContainer</span> <span class="n">activation</span><span class="p">;</span>
<span class="k">readonly</span> <span class="kt">int</span> <span class="n">outputChannels</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">ResNetBlock</span><span class="p">(</span><span class="kt">int</span> <span class="n">kernelSize</span><span class="p">,</span> <span class="kt">int</span><span class="p">[]</span> <span class="n">filters</span><span class="p">,</span>
<span class="n">PythonFunctionContainer</span> <span class="n">activation</span> <span class="p">=</span> <span class="k">null</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">activation</span> <span class="p">=</span> <span class="n">activation</span> <span class="p">??</span> <span class="n">tf</span><span class="p">.</span><span class="n">keras</span><span class="p">.</span><span class="n">activations</span><span class="p">.</span><span class="n">relu_fn</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">part</span> <span class="p">=</span> <span class="m">0</span><span class="p">;</span> <span class="n">part</span> <span class="p"><</span> <span class="n">PartCount</span><span class="p">;</span> <span class="n">part</span><span class="p">++)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">convs</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nf">Track</span><span class="p">(</span><span class="n">part</span> <span class="p">==</span> <span class="m">1</span>
<span class="p">?</span> <span class="n">Conv2D</span><span class="p">.</span><span class="nf">NewDyn</span><span class="p">(</span>
<span class="n">filters</span><span class="p">:</span> <span class="n">filters</span><span class="p">[</span><span class="n">part</span><span class="p">],</span>
<span class="n">kernel_size</span><span class="p">:</span> <span class="n">kernelSize</span><span class="p">,</span>
<span class="n">padding</span><span class="p">:</span> <span class="s">"same"</span><span class="p">)</span>
<span class="p">:</span> <span class="n">Conv2D</span><span class="p">.</span><span class="nf">NewDyn</span><span class="p">(</span><span class="n">filters</span><span class="p">[</span><span class="n">part</span><span class="p">],</span> <span class="n">kernel_size</span><span class="p">:</span> <span class="p">(</span><span class="m">1</span><span class="p">,</span> <span class="m">1</span><span class="p">))));</span>
<span class="k">this</span><span class="p">.</span><span class="n">batchNorms</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nf">Track</span><span class="p">(</span><span class="k">new</span> <span class="nf">BatchNormalization</span><span class="p">()));</span>
<span class="p">}</span>
<span class="k">this</span><span class="p">.</span><span class="n">outputChannels</span> <span class="p">=</span> <span class="n">filters</span><span class="p">[</span><span class="n">PartCount</span> <span class="p">-</span> <span class="m">1</span><span class="p">];</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">override</span> <span class="kt">dynamic</span> <span class="nf">call</span><span class="p">(</span>
<span class="kt">object</span> <span class="n">inputs</span><span class="p">,</span>
<span class="n">ImplicitContainer</span><span class="p"><</span><span class="n">IGraphNodeBase</span><span class="p">></span> <span class="n">training</span> <span class="p">=</span> <span class="k">null</span><span class="p">,</span>
<span class="n">IEnumerable</span><span class="p"><</span><span class="n">IGraphNodeBase</span><span class="p">></span> <span class="n">mask</span> <span class="p">=</span> <span class="k">null</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nf">CallImpl</span><span class="p">((</span><span class="n">Tensor</span><span class="p">)</span><span class="n">inputs</span><span class="p">,</span> <span class="n">training</span><span class="p">?.</span><span class="n">Value</span><span class="p">);</span>
<span class="p">}</span>
<span class="kt">object</span> <span class="nf">CallImpl</span><span class="p">(</span><span class="n">IGraphNodeBase</span> <span class="n">inputs</span><span class="p">,</span> <span class="kt">dynamic</span> <span class="n">training</span><span class="p">)</span> <span class="p">{</span>
<span class="n">IGraphNodeBase</span> <span class="n">result</span> <span class="p">=</span> <span class="n">inputs</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">batchNormExtraArgs</span> <span class="p">=</span> <span class="k">new</span> <span class="n">PythonDict</span><span class="p"><</span><span class="kt">string</span><span class="p">,</span> <span class="kt">object</span><span class="p">>();</span>
<span class="k">if</span> <span class="p">(</span><span class="n">training</span> <span class="p">!=</span> <span class="k">null</span><span class="p">)</span>
<span class="n">batchNormExtraArgs</span><span class="p">[</span><span class="s">"training"</span><span class="p">]</span> <span class="p">=</span> <span class="n">training</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">part</span> <span class="p">=</span> <span class="m">0</span><span class="p">;</span> <span class="n">part</span> <span class="p"><</span> <span class="n">PartCount</span><span class="p">;</span> <span class="n">part</span><span class="p">++)</span> <span class="p">{</span>
<span class="n">result</span> <span class="p">=</span> <span class="k">this</span><span class="p">.</span><span class="n">convs</span><span class="p">[</span><span class="n">part</span><span class="p">].</span><span class="nf">apply</span><span class="p">(</span><span class="n">result</span><span class="p">);</span>
<span class="n">result</span> <span class="p">=</span> <span class="k">this</span><span class="p">.</span><span class="n">batchNorms</span><span class="p">[</span><span class="n">part</span><span class="p">].</span><span class="nf">apply</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">:</span> <span class="n">batchNormExtraArgs</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">part</span> <span class="p">+</span> <span class="m">1</span> <span class="p">!=</span> <span class="n">PartCount</span><span class="p">)</span>
<span class="n">result</span> <span class="p">=</span> <span class="p">((</span><span class="kt">dynamic</span><span class="p">)</span><span class="k">this</span><span class="p">.</span><span class="n">activation</span><span class="p">)(</span><span class="n">result</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">result</span> <span class="p">=</span> <span class="p">(</span><span class="n">Tensor</span><span class="p">)</span><span class="n">result</span> <span class="p">+</span> <span class="n">inputs</span><span class="p">;</span>
<span class="k">return</span> <span class="p">((</span><span class="kt">dynamic</span><span class="p">)</span><span class="k">this</span><span class="p">.</span><span class="n">activation</span><span class="p">)(</span><span class="n">result</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">override</span> <span class="kt">dynamic</span> <span class="nf">compute_output_shape</span><span class="p">(</span><span class="n">TensorShape</span> <span class="n">input_shape</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">input_shape</span><span class="p">.</span><span class="n">ndims</span> <span class="p">==</span> <span class="m">4</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">outputShape</span> <span class="p">=</span> <span class="n">input_shape</span><span class="p">.</span><span class="nf">as_list</span><span class="p">();</span>
<span class="n">outputShape</span><span class="p">[</span><span class="m">3</span><span class="p">]</span> <span class="p">=</span> <span class="k">this</span><span class="p">.</span><span class="n">outputChannels</span><span class="p">;</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">TensorShape</span><span class="p">(</span><span class="n">outputShape</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">input_shape</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And the model itself is just a sequence of ResNet blocks with some pooling operations in-between +
a fully-connected (<a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense">Dense</a>
in TensorFlow) <a href="https://en.wikipedia.org/wiki/Softmax_function">softmax</a> layer
to output “probabilities”:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">static</span> <span class="n">Model</span> <span class="nf">CreateModel</span><span class="p">(</span><span class="kt">int</span> <span class="n">classCount</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">activation</span> <span class="p">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">keras</span><span class="p">.</span><span class="n">activations</span><span class="p">.</span><span class="n">elu_fn</span><span class="p">;</span>
<span class="k">const</span> <span class="kt">int</span> <span class="n">filterCount</span> <span class="p">=</span> <span class="m">8</span><span class="p">;</span>
<span class="kt">int</span><span class="p">[]</span> <span class="n">resNetFilters</span> <span class="p">=</span> <span class="p">{</span> <span class="n">filterCount</span><span class="p">,</span> <span class="n">filterCount</span><span class="p">,</span> <span class="n">filterCount</span> <span class="p">};</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">Sequential</span><span class="p">(</span><span class="k">new</span> <span class="n">Layer</span><span class="p">[]</span> <span class="p">{</span>
<span class="k">new</span> <span class="nf">Dropout</span><span class="p">(</span><span class="n">rate</span><span class="p">:</span> <span class="m">0.05</span><span class="p">),</span>
<span class="n">Conv2D</span><span class="p">.</span><span class="nf">NewDyn</span><span class="p">(</span><span class="n">filters</span><span class="p">:</span> <span class="n">filterCount</span><span class="p">,</span> <span class="n">kernel_size</span><span class="p">:</span> <span class="m">5</span><span class="p">,</span> <span class="n">padding</span><span class="p">:</span> <span class="s">"same"</span><span class="p">),</span>
<span class="n">Activation</span><span class="p">.</span><span class="nf">NewDyn</span><span class="p">(</span><span class="n">activation</span><span class="p">),</span>
<span class="k">new</span> <span class="nf">MaxPool2D</span><span class="p">(</span><span class="n">pool_size</span><span class="p">:</span> <span class="m">2</span><span class="p">),</span>
<span class="k">new</span> <span class="nf">ResNetBlock</span><span class="p">(</span><span class="n">kernelSize</span><span class="p">:</span> <span class="m">3</span><span class="p">,</span> <span class="n">filters</span><span class="p">:</span> <span class="n">resNetFilters</span><span class="p">,</span> <span class="n">activation</span><span class="p">:</span> <span class="n">activation</span><span class="p">),</span>
<span class="k">new</span> <span class="nf">ResNetBlock</span><span class="p">(</span><span class="n">kernelSize</span><span class="p">:</span> <span class="m">3</span><span class="p">,</span> <span class="n">filters</span><span class="p">:</span> <span class="n">resNetFilters</span><span class="p">,</span> <span class="n">activation</span><span class="p">:</span> <span class="n">activation</span><span class="p">),</span>
<span class="k">new</span> <span class="nf">ResNetBlock</span><span class="p">(</span><span class="n">kernelSize</span><span class="p">:</span> <span class="m">3</span><span class="p">,</span> <span class="n">filters</span><span class="p">:</span> <span class="n">resNetFilters</span><span class="p">,</span> <span class="n">activation</span><span class="p">:</span> <span class="n">activation</span><span class="p">),</span>
<span class="k">new</span> <span class="nf">ResNetBlock</span><span class="p">(</span><span class="n">kernelSize</span><span class="p">:</span> <span class="m">3</span><span class="p">,</span> <span class="n">filters</span><span class="p">:</span> <span class="n">resNetFilters</span><span class="p">,</span> <span class="n">activation</span><span class="p">:</span> <span class="n">activation</span><span class="p">),</span>
<span class="k">new</span> <span class="nf">MaxPool2D</span><span class="p">(),</span>
<span class="k">new</span> <span class="nf">ResNetBlock</span><span class="p">(</span><span class="n">kernelSize</span><span class="p">:</span> <span class="m">3</span><span class="p">,</span> <span class="n">filters</span><span class="p">:</span> <span class="n">resNetFilters</span><span class="p">,</span> <span class="n">activation</span><span class="p">:</span> <span class="n">activation</span><span class="p">),</span>
<span class="k">new</span> <span class="nf">ResNetBlock</span><span class="p">(</span><span class="n">kernelSize</span><span class="p">:</span> <span class="m">3</span><span class="p">,</span> <span class="n">filters</span><span class="p">:</span> <span class="n">resNetFilters</span><span class="p">,</span> <span class="n">activation</span><span class="p">:</span> <span class="n">activation</span><span class="p">),</span>
<span class="k">new</span> <span class="nf">MaxPool2D</span><span class="p">(),</span>
<span class="k">new</span> <span class="nf">ResNetBlock</span><span class="p">(</span><span class="n">kernelSize</span><span class="p">:</span> <span class="m">3</span><span class="p">,</span> <span class="n">filters</span><span class="p">:</span> <span class="n">resNetFilters</span><span class="p">,</span> <span class="n">activation</span><span class="p">:</span> <span class="n">activation</span><span class="p">),</span>
<span class="k">new</span> <span class="nf">ResNetBlock</span><span class="p">(</span><span class="n">kernelSize</span><span class="p">:</span> <span class="m">3</span><span class="p">,</span> <span class="n">filters</span><span class="p">:</span> <span class="n">resNetFilters</span><span class="p">,</span> <span class="n">activation</span><span class="p">:</span> <span class="n">activation</span><span class="p">),</span>
<span class="k">new</span> <span class="nf">AvgPool2D</span><span class="p">(</span><span class="n">pool_size</span><span class="p">:</span> <span class="m">2</span><span class="p">),</span>
<span class="k">new</span> <span class="nf">Flatten</span><span class="p">(),</span>
<span class="k">new</span> <span class="nf">Dense</span><span class="p">(</span><span class="n">units</span><span class="p">:</span> <span class="n">classCount</span><span class="p">,</span> <span class="n">activation</span><span class="p">:</span> <span class="n">tf</span><span class="p">.</span><span class="n">nn</span><span class="p">.</span><span class="n">softmax_fn</span><span class="p">),</span>
<span class="p">});</span>
<span class="p">}</span>
</code></pre></div></div>
<p>I also applied a 5% dropout to the input for regularization (the first in sequence):
it prevents model from memorizing exact lines of code from the training set by masking
different random characters out in different training iterations.</p>
<p>Part of the resulting model graph from TensorBoard:</p>
<p><img src="/images/NotCSharp-Model.png" alt="Part of neural network graph from TensorBoard" /></p>
<h2 id="training-the-network">Training the network</h2>
<p>So the model is built, training inputs are layed one after another in a giant byte array.
All that is left is to specify loss function (since we are classifying between several languages,
we will use <code class="language-plaintext highlighter-rouge">sparse_categorical_crossentropy</code>), and an optimizer (<code class="language-plaintext highlighter-rouge">Adam</code>). And start training:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">model</span><span class="p">.</span><span class="nf">compile</span><span class="p">(</span>
<span class="n">optimizer</span><span class="p">:</span> <span class="k">new</span> <span class="nf">Adam</span><span class="p">(),</span>
<span class="n">loss</span><span class="p">:</span> <span class="n">tf</span><span class="p">.</span><span class="n">keras</span><span class="p">.</span><span class="n">losses</span><span class="p">.</span><span class="n">sparse_categorical_crossentropy_fn</span><span class="p">,</span>
<span class="n">metrics</span><span class="p">:</span> <span class="k">new</span> <span class="kt">object</span><span class="p">[]</span> <span class="p">{</span> <span class="s">"accuracy"</span> <span class="p">});</span>
<span class="n">model</span><span class="p">.</span><span class="nf">build</span><span class="p">(</span><span class="n">input_shape</span><span class="p">:</span> <span class="k">new</span> <span class="nf">TensorShape</span><span class="p">(</span><span class="k">null</span><span class="p">,</span> <span class="n">Height</span><span class="p">,</span> <span class="n">Width</span><span class="p">,</span> <span class="m">1</span><span class="p">));</span>
<span class="n">model</span><span class="p">.</span><span class="nf">fit</span><span class="p">(</span><span class="n">@in</span><span class="p">,</span> <span class="n">expectedOut</span><span class="p">,</span>
<span class="n">batchSize</span><span class="p">:</span> <span class="k">this</span><span class="p">.</span><span class="n">BatchSize</span><span class="p">,</span> <span class="n">epochs</span><span class="p">:</span> <span class="k">this</span><span class="p">.</span><span class="n">Epochs</span><span class="p">,</span>
<span class="n">callbacks</span><span class="p">:</span> <span class="k">new</span> <span class="n">ICallback</span><span class="p">[]</span> <span class="p">{</span>
<span class="n">checkpointBest</span><span class="p">,</span>
<span class="n">tensorboard</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">);</span>
</code></pre></div></div>
<p>This also uses TensorFlow Keras callbacks:</p>
<ul>
<li><a href="https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint">ModelCheckpoint</a> - which saves model weights as it trains</li>
<li><a href="https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard">TensorBoard</a> -
which enables tracking the training progress using TensorBoard.</li>
</ul>
<p>Here you can see the training progress:</p>
<p><img src="/images/CSharp-NOT-TensorBoard.png" alt="Training progress on TensorBoard" /></p>
<p>The complete training code can be found <a href="https://github.com/losttech/Gradient-Samples/blob/1495521b6b8ba1b18c1167b838b08372bca8a4f3/CSharpOrNot/TrainCommand.cs#L36">here</a>.</p>
<h2 id="training-issues">Training Issues</h2>
<p>When I started with TensorFlow 1.10, after some training the model accuracy would drop to ~0.125,
which is the same, as if the model would pick random answer all the time. After spending
a couple of days debugging my code and tweaking the model to avoid potential weight infinities,
I figured out the issue was actually caused by a nasty bug in TensorFlow 1.10’s
<code class="language-plaintext highlighter-rouge">BatchNormalization</code> layer. So I had to install TensorFlow 1.14 alongside 1.10 and use it instead
(see <a href="https://github.com/losttech/Gradient#selecting-python-environment">how to</a>).</p>
<details>
<summary>XKCD debugging</summary>
<img src="https://4.bp.blogspot.com/-y-TH7oO-zOE/WHZdaHXLHpI/AAAAAAAA_Ts/SeF5gqOa1J4I2l6rywBYuZqZ04-nXH1PACLcB/s1600/xkcd_fuzzing_1137.png" />
</details>
<p>Also, the original model did not have the dropout layer right after the input, so it was overfitting,
until I added one.</p>
<h2 id="hooray">Hooray!</h2>
<p>After about 2 days of training on my GPU, I got a model with ~86% accuracy.
E.g. given a 64x64 block of text from a code file, it would recognize the language
correctly in ~86% cases, which is pretty good, considering C is similar to C++,
and they are both could be in <code class="language-plaintext highlighter-rouge">.h</code> header files. Also Java and C# copy each other.</p>
<p>As seen on the graphs above, the validation accuracy did not go up much after ~100M batches,
so I could have stopped training after ~20 hours. Also, the training code could be optimized by
using slightly larger batch sizes, and switching to 16-bit floats on GPUs that support them.</p>
<p>I published the trained weights to <a href="https://github.com/losttech/Gradient-Samples/releases/tag/csharp-or-not%2Fv1">GitHub</a>,
so you don’t have to train from scratch, if you just want to play with it.</p>
<h1 id="making-an-app-avalonia-ui">Making an app: Avalonia UI</h1>
<p>That was the easy part of Not C#: I followed Avalonia’s <a href="https://avaloniaui.net/docs/quickstart/create-new-project">official tutorial</a>.
A couple of Grids, TextBlocks and some event wiring, and the app was able to load
code files, and use <code class="language-plaintext highlighter-rouge">TextBox</code> cursor position to select the starting point
of a 64x64 block of characters.</p>
<p>It also used cross-platform <code class="language-plaintext highlighter-rouge">System.Drawing.Common</code> NuGet package to render a nice image,
that network would see as the input. It worked great on Windows out of the box,
but on Mac I had to install <code class="language-plaintext highlighter-rouge">mono-libgdiplus</code> following <a href="https://medium.com/@hudsonmendes/solved-system-drawing-netcore-on-mac-gdiplus-exception-c455ab3655a2">this advice</a>:</p>
<p><code class="language-plaintext highlighter-rouge">brew install mono-libgdiplus</code></p>
<blockquote>
<p>NOTE: I had to use TensorFlow 1.14 for inference as well.</p>
</blockquote>
<p>Also remember, that any input to the pretrained model must be processed the same way
the training data was: Unicode characters replaced, and whitespace set to 255.</p>
<p><img src="/images/ItsCSharp.png" alt="It's C#!" /></p>
<p><strong>Look, it is C#!</strong></p>
<h1 id="conclusion">Conclusion</h1>
<p>Not C# demonstrates how to design and train a deep convolutional neural network
with C# and TensorFlow. High-level Keras API is used to describe the model, and
some OOP helps to abstract its building blocks away.</p>
<p>The pre-trained network can be used in a cross-platform UI app
to load text files, and classify the programming language a block of text is written in.</p>
<p>The full up-to-date code of Not C# can be found on
<a href="https://github.com/losttech/Gradient-Samples/tree/master/CSharpOrNot">GitHub</a>.
Also, <a href="https://github.com/losttech/Gradient-Samples/releases/tag/csharp-or-not%2Fv1">pretrained weights for v1</a>.</p>In Silicon Valley season 4, Jian-Yang builds an AI app that identifies pictures of hotdogs. Today I am going to make a similar app to identify C# from code fragments (in case you forget how C# looks like!). Look at this Python trying to pretend to be your favorite language: var = await.add(item) switch(hello) Of course the whole thing from building and training a deep convolutional network to the cross-platform UI will be written in .NET + TensorFlow. Advanced-level tutorial on training a deep convnet to classify language from code fragmentsIndenting contents of namespace in C#2019-09-18T00:00:00+00:002019-09-18T00:00:00+00:00https://lostmsu.github.io/Indent-Namespace<p>C# occupies a lot of horizontal space. Pretty much every file
has a <code class="language-plaintext highlighter-rouge">namespace</code>, which by default adds 4 characters, and
then another 4 characters from the actual <code class="language-plaintext highlighter-rouge">class</code> you are
trying to define.</p>
<p>Take a look at this GitHub screenshot on my laptop for
a simple pull request:
<a href="https://github.com/dotnet/roslyn/pull/38755/files#diff-4acadf01a310c77a34de6ba7c5073347"><img src="/images/NamespaceIndentation.png" alt="C# diff wrapping on GitHub" /></a></p>
<p>The getters and setters for the property had to be wrapped,
because the lines are too long.</p>
<p>Since I had some spare time during today’s ML developers
summit in Google Kirland, I decided to fix it. And it ended up
pretty easy to do.</p>
<p>After cloning Roslyn and firing up
Roslyn.sln in Visual Studio, I quickly found a property named
<code class="language-plaintext highlighter-rouge">IndentSwitch-bla-bla</code> in the code. (I knew what to search
for, as there’s an option already for indenting <code class="language-plaintext highlighter-rouge">switch</code>
contents in Visual Studio).</p>
<p>Searching for references to this property gave back only
about 10 places, of which only one was an actual
implementation in the Roslyn formatting engine. As it
turned out, I only needed to <a href="https://github.com/dotnet/roslyn/pull/38755/files#diff-b12f01a1a15f45c02b23b989600b7cb7">add a simple <code class="language-plaintext highlighter-rouge">if</code> branch</a>
that skips indent procedure for a certain AST node type:
<code class="language-plaintext highlighter-rouge">NamespaceDeclarationSyntax</code> if the option is turned off.
The rest of the changes are unit tests, and view model
construction to actually show a setting in Visual Studio.</p>
<p>All that took under an hour. Check out the result: <a href="https://github.com/dotnet/roslyn/pull/38755">pull request</a>.</p>
<p>Once it is released with Visual Studio, I am planning to
switch that setting off on my machines and in <code class="language-plaintext highlighter-rouge">.editorconfig</code>.</p>C# occupies a lot of horizontal space. Pretty much every file has a namespace, which by default adds 4 characters, and then another 4 characters from the actual class you are trying to define.The Mercurial Archive2019-08-27T00:00:00+00:002019-08-27T00:00:00+00:00https://lostmsu.github.io/Mercurial-Archive<p>How often do you look at the code you wrote ~10 years ago? Bitbucket announced <a href="https://news.ycombinator.com/item?id=20745393">Sunsetting Mercurial Support</a>, including complete removal of all repositories using that source control system. So naturally, as one of the unfortunate souls who initially picked HG over Git, I checked out what I had there.</p>
<p><img src="https://law.utexas.edu/wp-content/uploads/sites/31/2016/02/DSC00066__Small_.jpg" alt="Some Texas library" /></p>
<p>Some projects successfully migrated to GitHub or VSO, but there were still quite a few, which I did not touch in years. Several of them the naive me from 2010 even considered to have commercial potential, so they were private repositories. Soon, of course, I will share them all (but won’t tell you which is which).</p>
<h1 id="alternate-f"><a href="https://bitbucket.org/lost/alternate-f/">Alternate F#</a></h1>
<p>Where you can find my attempt to add support for <a href="https://en.wikipedia.org/wiki/Type_class">typeclasses and higher-kinded polymorphism</a> to F#, which even worked at least in part, if you believe <a href="https://bitbucket.org/lost/alternate-f/commits/403515e81a0c075a6eec9b7e9cb233d702d46c03">this commit message</a>.</p>
<p>Attempts to do it were actually preceded by a pure library implementation, which is briefly discussed <a href="https://fpish.net/topic/None/59738">here</a>.</p>
<p>In the experiment, I wanted to get the feature in with minimal changes to the compiler to make a proof of concept, as the F# compiler source is very hard to follow. So the syntax was suboptimal, however quite readable (from the <a href="https://bitbucket.org/lost/alternate-f/commits/403515e81a0c075a6eec9b7e9cb233d702d46c03">abovementioned commit</a>):</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c1">/// This is 2D vector, that can print itself via Show<'a>.Print</span>
<span class="k">type</span> <span class="nc">Vector2D</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="o">>(</span><span class="n">x</span><span class="p">:</span> <span class="k">'</span><span class="n">a</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="k">'</span><span class="n">a</span><span class="p">)</span> <span class="p">=</span>
<span class="n">typeclass</span> <span class="n">show</span> <span class="p">=</span> <span class="nc">Show</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="p">></span>
<span class="n">typeclass</span> <span class="n">t</span> <span class="p">=</span> <span class="nc">Repeatable</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="p">></span>
<span class="k">member</span> <span class="n">this</span><span class="p">.</span><span class="nc">Times</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="p">=</span>
<span class="nc">Vector2D</span><span class="p">(</span><span class="n">x</span><span class="p">.</span><span class="nc">Times</span><span class="p">(</span><span class="n">n</span><span class="o">),</span> <span class="nc">Times</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">n</span><span class="o">))</span>
<span class="k">member</span> <span class="n">this</span><span class="p">.</span><span class="nc">Print</span><span class="bp">()</span> <span class="p">=</span>
<span class="nc">Print</span> <span class="n">x</span>
</code></pre></div></div>
<p>Note, that GitHub’s code highlighter does not recognize <code class="language-plaintext highlighter-rouge">typecalss</code> as a keyword. It is because F# compiler still does not have that feature.</p>
<p>Similarly to Go’s interfaces, you’d never have to explicitly state, that <code class="language-plaintext highlighter-rouge">'a</code> actually implements <code class="language-plaintext highlighter-rouge">Show</code>. The implementation would be wired at runtime, and for the compile time a special diagnostic would actually check, that <code class="language-plaintext highlighter-rouge">'a</code> you passed has all the necessary members.</p>
<p>Unfortunately, I stopped using F#, because the tooling around it is quite far behind what ReSharper and now even Roslyn do for C#. The last commit to that repository was on April 1st, 2012.</p>
<h1 id="codeblock"><a href="https://github.com/lostmsu/CodeBlock">CodeBlock</a></h1>
<p>This project actually found its home on GitHub, because some day in 2014 a person pinged me about using <a href="https://github.com/lostmsu/LLVM.NET">LLVM binding for C#</a>, which I had developed for it, so I migrated both.</p>
<p>CodeBlock was my <a href="https://llvm.org">LLVM</a> playground. The project was supposed to be an implementation of .NET runtime, written mostly in .NET itself rather than C, using LLVM as a backend. Quite similar to <a href="https://github.com/dotnet/coreclr">CoreCLR</a> in nature. I wrote most of it during my student years.</p>
<p>In this example <code class="language-plaintext highlighter-rouge">assertTest</code> would compile & execute code in between <code class="language-plaintext highlighter-rouge"><@</code> and <code class="language-plaintext highlighter-rouge">@></code>, e.g. <code class="language-plaintext highlighter-rouge">FactorialMethod</code>:</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="k">member</span> <span class="nc">FactorialMethod</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="p">=</span>
<span class="k">let</span> <span class="k">mutable</span> <span class="n">result</span> <span class="p">=</span> <span class="mi">1</span>
<span class="k">let</span> <span class="k">mutable</span> <span class="n">i</span> <span class="p">=</span> <span class="mi">2</span>
<span class="k">while</span> <span class="n">i</span> <span class="o"><=</span> <span class="n">n</span> <span class="k">do</span>
<span class="n">result</span> <span class="p"><-</span> <span class="n">result</span> <span class="p">*</span> <span class="n">i</span>
<span class="n">i</span> <span class="p"><-</span> <span class="n">i</span> <span class="o">+</span> <span class="mi">1</span>
<span class="n">result</span>
<span class="p">[<</span><span class="nc">TestMethod</span><span class="p">>]</span>
<span class="k">member</span> <span class="n">this</span><span class="p">.</span><span class="nc">FactorialNeg</span><span class="bp">()</span> <span class="p">=</span>
<span class="p">[</span> <span class="p">-</span><span class="mi">2</span><span class="p">;</span> <span class="p">-</span><span class="mi">1</span><span class="p">;</span> <span class="nn">Int32</span><span class="p">.</span><span class="nc">MinValue</span> <span class="p">]</span>
<span class="p">|></span> <span class="nn">List</span><span class="p">.</span><span class="n">iter</span><span class="p">(</span><span class="k">fun</span> <span class="n">n</span> <span class="p">-></span> <span class="n">assertTest</span> <span class="o"><@</span> <span class="nn">PrimitiveOps</span><span class="p">.</span><span class="nc">FactorialMethod</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="o">@>)</span>
<span class="p">[<</span><span class="nc">TestMethod</span><span class="p">>]</span>
<span class="k">member</span> <span class="n">this</span><span class="p">.</span><span class="nc">FactorialZero</span><span class="bp">()</span> <span class="p">=</span>
<span class="n">assertTest</span> <span class="o"><@</span> <span class="nn">PrimitiveOps</span><span class="p">.</span><span class="nc">FactorialMethod</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">@></span>
</code></pre></div></div>
<p>It was using Mono.Cecil to read method bodies bytecode that .NET uses (similar to Java bytecode), and then use LLVM 3’s JIT APIs to convert it to native x64 or x86 code before execution.</p>
<p>As far as I remember (and judging by other contents of <a href="https://github.com/lostmsu/CodeBlock/tree/master/CodeBlock.UnitTests">the unit tests folder</a>), it went as far as to have most of unmanaged CLR features working, including pointers and <code class="language-plaintext highlighter-rouge">struct</code> generic types. Garbage collector would be the next step and some of the <a href="https://github.com/lostmsu/CodeBlock/tree/master/CodeBlock.BasicObjectModel">“object model”</a> was already defined. The last meaningful commit to that repository was on August 7th, 2013.</p>
<h1 id="corecloud"><a href="https://github.com/lostmsu/CoreCloud">CoreCloud</a></h1>
<p>I am sure some readers here have seen the infamous show “Silicon Valley”. In 2017 season it was revealed, that Pied Piper is to build what Peter Gregory planned all along: a truly decentralized Internet. Well, CoreCloud was in some sense a 2010 take on that dream with trustless compute, and P2P IPv6 which was surely supposed to take over the world by 2015.</p>
<p>The goal of the project was to make it super simple to run decentralized trustless computations in .NET (which, by the way, had all the necessary components back then). Here is how it worked: first, you would declare a class, whose code you want to run on many machines, and its interface like this:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">PrimeTest</span><span class="p">:</span> <span class="n">IPrimeTest</span>
<span class="p">{</span>
<span class="k">public</span> <span class="kt">bool</span><span class="p">[]</span> <span class="nf">Test</span><span class="p">(</span><span class="kt">int</span> <span class="n">start</span><span class="p">,</span> <span class="kt">int</span> <span class="n">count</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">nums</span> <span class="p">=</span> <span class="n">Enumerable</span><span class="p">.</span><span class="nf">Range</span><span class="p">(</span><span class="n">start</span><span class="p">,</span> <span class="n">count</span><span class="p">);</span>
<span class="k">return</span> <span class="n">nums</span><span class="p">.</span><span class="nf">Select</span><span class="p">(</span><span class="n">IsPrime</span><span class="p">).</span><span class="nf">ToArray</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">internal</span> <span class="kt">bool</span> <span class="nf">IsPrime</span><span class="p">(</span><span class="kt">int</span> <span class="n">num</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">num</span> <span class="p"><</span> <span class="m">2</span><span class="p">)</span> <span class="k">return</span> <span class="k">false</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="p">=</span> <span class="m">2</span><span class="p">;</span> <span class="n">i</span> <span class="p"><=</span> <span class="n">Math</span><span class="p">.</span><span class="nf">Sqrt</span><span class="p">(</span><span class="n">num</span><span class="p">);</span> <span class="n">i</span><span class="p">++)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">num</span> <span class="p">%</span> <span class="n">i</span> <span class="p">==</span> <span class="m">0</span><span class="p">)</span> <span class="k">return</span> <span class="k">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="k">true</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">[</span><span class="n">ServiceContract</span><span class="p">]</span>
<span class="k">public</span> <span class="k">interface</span> <span class="nc">IPrimeTest</span>
<span class="p">{</span>
<span class="p">[</span><span class="n">OperationContract</span><span class="p">]</span>
<span class="kt">bool</span><span class="p">[]</span> <span class="nf">Test</span><span class="p">(</span><span class="kt">int</span> <span class="n">start</span><span class="p">,</span> <span class="kt">int</span> <span class="n">count</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Then you would just create an instance of this class, and call its <code class="language-plaintext highlighter-rouge">Test</code> method in parallel like this:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">const</span> <span class="kt">int</span> <span class="n">ChunkSize</span> <span class="p">=</span> <span class="m">1024</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">primeTest</span> <span class="p">=</span> <span class="n">ParallelObject</span><span class="p"><</span><span class="n">IPrimeTest</span><span class="p">>.</span><span class="n">Create</span><span class="p"><</span><span class="n">PrimeTest</span><span class="p">>();</span>
<span class="kt">var</span> <span class="n">results</span> <span class="p">=</span> <span class="k">new</span> <span class="n">ConcurrentDictionary</span><span class="p"><</span><span class="kt">long</span><span class="p">,</span> <span class="kt">bool</span><span class="p">[</span><span class="k">]></span><span class="p">();</span>
<span class="n">Parallel</span><span class="p">.</span><span class="nf">For</span><span class="p">(</span><span class="m">0</span><span class="p">,</span> <span class="m">1024</span><span class="p">,</span> <span class="n">chunk</span>
<span class="p">=></span> <span class="n">results</span><span class="p">[</span><span class="n">chunk</span><span class="p">]</span> <span class="p">=</span> <span class="n">primeTest</span><span class="p">.</span><span class="nf">Test</span><span class="p">(</span><span class="n">chunk</span><span class="p">*</span><span class="n">ChunkSize</span><span class="p">,</span> <span class="n">ChunkSize</span><span class="p">));</span>
</code></pre></div></div>
<p>Behind the scenes CoreCloud would query a <a href="https://docs.microsoft.com/en-us/dotnet/api/system.net.peertopeer">peer-to-peer network</a> of people running CoreCloud on their machines, pick some of them, transmit the code of <code class="language-plaintext highlighter-rouge">PrimeTest</code>, and ask them to instantiate it for you in a sandbox (now that .NET feature is <a href="https://docs.microsoft.com/en-us/dotnet/framework/misc/how-to-run-partially-trusted-code-in-a-sandbox">deprecated</a> 😢). <code class="language-plaintext highlighter-rouge">primeTest</code> here will actually be a proxy generated by CoreCloud on the fly for <code class="language-plaintext highlighter-rouge">IPrimeTest</code>. Every call to <code class="language-plaintext highlighter-rouge">Test</code> in the <code class="language-plaintext highlighter-rouge">Parallel.For</code> loop CoreCloud would pick one of the remote instances of <code class="language-plaintext highlighter-rouge">PrimeTest</code> it created, and forward it there.</p>
<p>The project was to be demoed on a graphics engine, which would use nearby computers to deliver real-time full HD ray-tracing-based rendering in a very similar manner with <code class="language-plaintext highlighter-rouge">Engine.TraceRays</code> replacing <code class="language-plaintext highlighter-rouge">PrimeTest.Test</code> in the example above. The last commit to the original Mercurial repository was on May 8th, 2011.</p>
<h1 id="to-be-continued">To be continued…</h1>How often do you look at the code you wrote ~10 years ago? Bitbucket announced Sunsetting Mercurial Support, including complete removal of all repositories using that source control system. So naturally, as one of the unfortunate souls who initially picked HG over Git, I checked out what I had there.