Posts on redgeoff

My first trading models knew which companies would survive

Mon, 05 Oct 2026 15:38:00 -0700

Image credit: author, drawn with code

My first live trading bot started trading real money on December 2, 2024. The models behind it had been trained on years of stock-market history, and every company in that history had something in common: it was in the S&P 500 on the day I trained the model.

That sounds harmless until you think about what it leaves out. A company that was in the index in 2016 and went bankrupt in 2019 wasn’t in the training data at all, and a company that joined the index in 2025 was in it all the way back. In other words, the models learned from a version of the market where nothing ever failed and every stock was already a future winner.

That was a leak, and it went live. It wasn’t the only one either.

If you haven’t built one of these systems, here’s what I mean by a leak. You can’t test a trading model by waiting ten years to see how it does, so instead you backtest it: replay history, let the model pick stocks as if it were living through each date, and score those picks against what actually happened next. A leak is any information that sneaks into that replay which the model couldn’t have had on the date in question. It doesn’t have to be anything as obvious as tomorrow’s price. Knowing which companies will still be around in five years is enough, and the result is a backtest that looks better than anything you could have done for real.

The standard defence is to train walk-forward. For each rebalancing date (the days the portfolio is reshuffled, roughly monthly for me), the model trains only on data from before that date, picks its stocks, holds them until the next rebalancing date, and then the whole thing steps forward and repeats. That way, no single model is ever scored on data it was trained on. Walk-forward is necessary, but as you’ll see, every leak in this post got through it, because each one smuggled the future in somewhere other than the training dates: the list of stocks, the shape of the data, or the data itself, through prices adjusted the wrong way or figures dated before anyone could have known them.

So, this post is about the leaks I shipped, the ones I caught in research before they shipped, and the times an AI was sure it had found a leak and was wrong. Every one of them is a reason that backtests are hard to trust, including your own. Along the way I’ve ended up with one rule for all three: you can’t reason your way to the answer. You need a test that makes a leak show up as a number.

Survivors

What gave it away was a backtest that was too good. I tune the model by running hyperparameter searches, where I backtest hundreds of versions of the model with different settings and compare them. In August 2025, one of those runs came back with a 2020 that was simply impossible, and most of that year’s profit came from a single holding. When I went to look it up, neither Yahoo Finance nor Google Finance had any prices for its ticker before 2021, and my note from the investigation reads: “Perhaps the symbol was renamed?”

It had been. It was Chesapeake Energy, which had been dropped from the S&P 500 in 2018, did a 1-for-200 reverse split in April 2020, went bankrupt later that year, and eventually merged, took on a new name and rejoined the index in 2025. My backtest was holding it in 2020, two years after it had left the index, because the training code only knew about the index as it stood today. (Keep that reverse split in mind, as it shows up again later in this post.)

Fixing survivorship bias turned out to be mostly a data problem. First, I needed the S&P 500’s membership for every day of the training period, not just the current list. I built this from the public record of index changes and had a couple of AI models double-check the list. Then I needed price history for the companies that had left the index, many of which no longer trade. Finally, the training loop had to resolve the set of stocks it was allowed to pick from (its universe) date by date, so that a model trained on 2016 data only ever saw companies that were actually in the index in 2016.

When the fix first ran, the backtest fell apart. The note I left myself in the pull request was, in all caps, “THE ANNUALIZED RETURN IS NEAR 0!!!” That’s what removing a leak is supposed to do, and it’s a very unpleasant thing to watch.

The fix reached the production training path in late September 2025, about ten months after the first live trade.

There was one experiment from that period that I’m still glad I ran. I wondered whether a small, controlled leak might be acceptable: train the first stretch of the model on a later snapshot of the index, so that it learned from today’s large companies, and then walk forward point-in-time from there. It gave no reliable improvement. My conclusion at the time was that a model trained on a fixed list of companies never sees a stock enter or leave the index, which leaves it less prepared for the real index, where that happens all the time.

Unfortunately, that wasn’t the end of it. A year later, in October 2026, a copy of the equal-weight S&P 500 that an agent and I had rebuilt from my research data beat the real equal-weight index fund by a margin that a copy has no business beating it by. The two moved almost in lockstep, so the copy was tracking the right thing, but it was consistently doing better. That’s the same “too good” signature, and it turned out to be survivorship again, coming from two places.

The first was the membership list itself. The public record of index changes I’d built it from is sparse before about 2007, so the early lists were short and slightly wrong. To check, we rebuilt membership from the holdings of an S&P 500 index fund, which has to file its complete list of holdings with the SEC. Against those filings, my list was 74 companies short in 2001 and still 5 short in 2009, and it even contained a few companies that hadn’t gone public yet. The AI models I’d asked to double-check the list in 2025 hadn’t caught any of this.

How many S&P 500 members my list knew about, against the holdings SPY filed with the SEC. The gap is the least my 2025 fix still left out, since the list also held a few companies that hadn’t gone public yet.

The second was prices. Many of the companies that failed outright, like Lehman Brothers, Bear Stearns, Enron and WorldCom, have no price data in either of my data sources, so even with a correct list they can never be picked or held in a backtest.

This one mostly affects my long-history research backtests rather than the live system, since the data the live configuration was chosen from covers 99.6% of index members. But it means that for my backtests before about 2016, the honest reading is “probably an upper bound”, and fixing it properly needs a data source that still sells prices for companies that no longer exist.

So, What Did It Cost?

I don’t know, and I think the reason is worth explaining.

First, a correction to the intuition. A live model can’t see the future, because on the day it makes a prediction there’s no future data to see. Today’s S&P 500 list is the correct list for today. So, the live models were never peeking at tomorrow’s prices. The damage was in two other places:

The backtests I used to choose the model and its settings were inflated by the leak, so the configuration that went live was selected on a flattering number.
The live model had been trained on a history with no failures in it, so whatever it had learned, it had learned from survivors.

Putting a price on that requires comparing what the model expected with what the account actually did, period by period, and in 2025 I had no mechanism for tracking that gap. Moreover, the model worked from end-of-day closing prices while the orders filled the next morning, so any gap I computed mixed model error with hours of ordinary market movement.

That was a consequence of how the early system worked. It ran at a fairly leisurely pace after the market closed: download the day’s prices, retrain the model, decide what to buy and sell, and queue the orders for the next morning. Since then, a lot of optimization and engineering has gone into doing all of that during the trading day. The system pulls a price snapshot at a fixed time in the morning, trains the model on it and starts trading, with a median of about 12 minutes from the snapshot to the first fill. For bots that run this way, the model’s prices and the fill prices are now minutes apart instead of a night apart.

There are also now reports that split a bot’s live result into what the model predicted, what the portfolio’s composition delivered and what execution cost. When I finally ran those reports across the live history in September 2026, every period I analyzed had run without a fixed price time, so the execution numbers came with a caveat that they couldn’t really be read as execution at all. The measurement that would have priced the leak simply didn’t exist yet.

The Leak the LLMs Couldn’t Find

Survivorship was actually the second leak to reach production. The first was subtler, and it’s where most of the testing in the rest of this post came from.

In March 2025, while analyzing the returns of a newer model, I started to worry that there was leakage in the older one. So, I compared two ways of training the same model. One trained incrementally, one rebalancing period at a time, exactly as it would run live and exactly as walk-forward requires. The other trained over the whole period at once. Given the same data and the same settings, they should have produced the same results, but the annualized returns were different.

I asked the LLMs I was working with to explain the difference and they couldn’t. The pull request I opened to debug it says so directly: “The LLMs appear to be unable to determine why the annualized returns for leakage_check.py and train_and_predict.py differ. Let’s roll up our sleeves and figure this out.”

Rolling up my sleeves meant writing a logger. I logged every call to fit() and predict(), along with the shape and columns of every input, to a JSON file without timestamps, ran both paths and diffed the files. Then I shortened the run until I had the smallest period that still showed a difference and stepped through it in the debugger.

The difference was in the shape of the training matrix. Which columns it had, and which rows survived a filter for missing values, depended on the data loaded for the whole run rather than just the training window. When training all at once, the model for an early period was shaped by companies whose data only began years later. When training incrementally, it didn’t know those companies existed yet. Neither path was reading a future price directly, but the future was deciding what the training data looked like, and that turned out to be enough to change the results.

The comparison that found the leak became a test: train incrementally on part of the data, continue on the rest, and the picks must be bit-for-bit identical to training on all of it from scratch. It started as a script that I ran by hand, then became the code path that production uses to train, and it now runs every week in the automated test suite. If I could only keep one test from this whole project, it would be this one.

The Fix That Switched Off Most of the Model

Unfortunately, the March 2025 refactor fixed the leak and broke something else, and nothing noticed for thirteen months.

Near the end of the data-preparation step, the refactored code selected the training columns by matching column names against a list of ticker symbols. The raw return column for each stock is named after its ticker, so it matched. However, the engineered features (moving averages and volatility measures built from those returns) have the ticker as a prefix rather than as the whole name, so they didn’t match and were dropped. A later step, which lines up every iteration’s columns with the model’s expected list of features, then quietly put them back, filled with zeros.

The model ran, the tests passed and the features were all present in the matrix. Every one of them was zero, and that version went into production along with the leak fix.

It finally came out in April 2026, while an LLM and I were wiring a new family of features into the training pipeline. Earlier that same day we had found a different silent drop, where three new settings were decoded from the model’s configuration and then never passed to the model. When we looked at how much the trained models relied on each input (their feature importance), every engineered feature had an importance of exactly zero.

The fix went in behind a flag so that it could be compared with the old behaviour, and a week later it became the default. If you read my last post, you may recall a change that did nothing, where two runs came back bit-for-bit identical. This is the same smell from a different angle, and I now treat an exact zero with the same suspicion as an exact match.

Corporate Actions Keep Coming Back

Most of the leaks I found in 2026 came from the same place: splits, reverse splits and dividends, which are known as corporate actions.

A quick primer, since everything below depends on it. A raw price is what a stock actually traded at on a given day. When a company does a 20-for-1 split, every share becomes 20 shares worth a twentieth as much, so the raw price drops 95% overnight even though no one lost a cent. A reverse split is the opposite: 200 shares become one, and the raw price jumps 200-fold. An adjusted price rewrites the history before each event so that the series is continuous, e.g. by dividing every price before a 20-for-1 split by 20, and it does something similar for dividends. Neither series is wrong. They answer different questions, and the bugs below all came from asking one series a question meant for the other.

I told the stock-dividend story in my last post, where a test asserted a data convention that never occurs in practice. Here are three more.

The Model Learned to Short Stock Splits

In a long/short research line, which buys some stocks and bets against others by shorting them, so that it profits when they fall, one data source’s features were computed from raw prices rather than adjusted ones. So, when Amazon split 20:1 in June 2022, the raw series showed a one-day move of about −95%. Alphabet’s 20:1 split about six weeks later looked the same, and Tesla’s 3:1 split read as −67%.

A −95% day dominates any ranking of stocks against each other, and the model’s momentum-style features, which look at how much a stock has risen or fallen recently, carried it for one to three months afterwards. The model learned that stocks with that shape made good shorts, but they weren’t failing companies. They had just split.

A Copilot agent found it while comparing the same strategy across two data vendors. One vendor’s prices came pre-adjusted, the other’s feature panel had been built from raw prices, and the two short sides disagreed by far more than data noise should explain. A simple count shows the problem: in a 2019 sample, 9.7% of the symbols in the raw panel had a single-day move bigger than 50%, against 6.1% in the other vendor’s adjusted data.

What I find interesting is why raw prices had been chosen in the first place. Adjusted prices rewrite the past, e.g. a 2022 split changes the adjusted price for 2018. That looks exactly like lookahead, so raw prices seemed like the safe choice. However, these features were ratios: a price today divided by a price from some days ago. For any window that doesn’t straddle a split, the split factor cancels out and raw and adjusted prices give identical values. For a window that does straddle a split, the adjusted ratio is the true economic return and the raw one is an artifact. In other words, the thing that looked like lookahead wasn’t, and the thing chosen to avoid it was the bug.

Amazon’s 2022 split as raw and adjusted prices. The raw one-month return reads as a 95% crash for a month: the shape the model learned to treat as a good short.

Ratios Want Adjusted, Levels Want Raw

The mirror image turned up about a month later. The same research line had a minimum-price filter so that it wouldn’t trade anything under $10, and it was being applied to adjusted prices.

After a large reverse split, adjustment inflates a stock’s earlier history. A shell company that really traded at a fraction of a cent, and then did a reverse split of a few hundred to one, ends up with an adjusted history of tens to thousands of dollars. So, those names passed the $10 floor, were trained on and were shorted, and the backtest booked wins on positions that nobody could have actually held. Claude found this one while investigating short-side results that still looked wrong after the earlier fixes. Gating on the raw price instead took the number of candidate instances priced under $10 from 11,194 to zero.

The rule I took from the pair is that a ratio wants adjusted prices, because it’s asking how much a holding changed, while a price level wants raw prices, because it’s asking what the stock actually traded at. Neither series is a safe default.

After four layers of price corruption had been removed from that research line, its log summed it up plainly: the earlier positive results had been “largely the corruption we removed.”

The Screen in Front of Everything Was Leaking

This last one is the one I find most instructive, because it was in the tool I use to decide what is worth testing in the first place.

I screen new signal ideas (candidate inputs for the model) with a quick check that runs in seconds, before committing to a hyperparameter search that runs for days. It asks a simple question: on each past date, did ranking stocks by this signal put them in roughly the right order of what they did next? In September 2026, an agent using that screen to check a textbook momentum signal for another project found that the screen itself had two problems.

The first was that it loaded prices through a function whose adjusted flag defaults to False, and which returns its price column under the key "Adj Close" either way. The variable was named for adjusted prices, the column was named for adjusted prices and every statistic downstream said adjusted, but the data was raw.

The second was that it built its universe once, for the whole run, from every company that had ever been in the index. On the dates checked, 42 to 47% of the names it was ranking on a given date weren’t index members on that date. Some joined later, which is lookahead, since companies are added to the index because they grew. Others had left years before.

The tell had been sitting in the documentation for months, as a statistic with a note beside it saying “investigate”. It traced back to Chesapeake’s 1-for-200 reverse split in April 2020, the same reverse split from the survivorship investigation a year earlier. Here, raw prices turned it into a gain of more than 10,000%, and it was counted twice because the universe carried both the old and the new ticker. The problem also generalizes. Reverse splits tend to happen to companies in trouble, and companies in trouble are exactly what sits at the bottom of a ranking by recent performance, so any signal that sorts on losers is contaminated in the same direction.

Fortunately, the decisions I had made with that screen were based on rank correlation, which only asks whether the stocks came out in the right order, not by how much, so a few enormous outliers barely move it. When the agent recomputed a signal on corrected data, its rank correlation hardly changed, so those decisions probably still stand. Two other columns in the screen’s output didn’t survive. A leak doesn’t always invalidate everything it touches, and most of the work is finding out which numbers it actually reached.

Time Has Two Dates

Point-in-time doesn’t just mean the date on the row.

Some data describes a period but only becomes public later. One data vendor only keys its quarterly company financials by the end of the quarter they describe. A company’s December quarter isn’t public until its annual report is filed, around the start of March for the largest companies, and the vendor then needs time to pick it up. A reporting lag of 45 trading days after the period end made those numbers “known” a week or two before the vendor would actually have had them. That setting had already stood out in my hyperparameter search results, and the vendor’s own documentation settled it, as the period end is the only date it provides. So, I removed the option.

EDGAR, the SEC’s filing database, does record the filing date, so when I later needed historical share counts, I keyed them on the date they were filed, typically about five weeks after the quarter they describe. Keying on the period end would have used numbers that nobody could have known yet.

One vendor field had no code bug at all. An earnings-surprise figure (how far a company’s reported earnings beat or missed what analysts expected) was being computed against the analysts’ average forecast as it stood after later revisions, rather than as it stood on the day the earnings came out. The fix was to rebuild it from point-in-time estimate snapshots, which meant dropping every year before the snapshots begin and about a fifth of the symbols, which had no snapshots at all. Sometimes an honest fix makes your dataset smaller.

The False Alarms

Leakage can also go wrong in the other direction.

In August 2025, in the middle of the survivorship work, two issues drafted with an AI’s help claimed there was lookahead in my volume filter and in how the code decided which symbols were available. Both read convincingly. When I traced the dates, though, the rebalancing decision happens at the end of the window the code was reading, so all of that data was already known. My reply on the first one starts, “Actually, I believe we DO NOT have a lookahead bias.”

A year later, an agent flagged code that was, in its words, “target-leakage-shaped”: the training target also appears among the input features. It does, and at a glance it looks alarming. However, tracing every date boundary showed no lookahead, as the training window ends where the test window begins and returns are measured forward from the pick date. The agent’s follow-up claim, that the model must therefore be a simple momentum ranker, didn’t survive a measurement either. That’s two confident answers, each corrected by measuring rather than arguing.

The repo’s instruction file for agents now says that any leakage claim must name the specific date boundary that it violates and cite the code path, and those boundaries are pinned by tests named test_no_lookahead_*.

Who Found What?

When I went back through the issues and pull requests to write this post, the pattern was clearer than I’d remembered. I found the early leaks, the ones that reached production, from numbers that were too good or didn’t match: an impossible year and two training paths that disagreed. The 2026 leaks were mostly found by agents, and almost never while they were looking for leaks. They turned up while we were wiring new features, building a new test harness, comparing two data vendors or checking a rebuilt index against the real one, with me reviewing what came back. The agents also raised most of the false alarms.

The tests mostly arrived after the leak that justified them. I don’t think that’s a failure. A test written before you’ve been bitten is a guess about where the bugs are, whereas a test written afterwards is a record of where they were.

Making Leaks Show Up as Numbers

Here’s what actually caught things, roughly in the order that I’d add them to a new project:

Invariance. A result for a past window must not depend on the future. Training incrementally must match training all at once, and moving a run’s end date must not change an earlier window. The March 2025 leak broke exactly this, and so did a similar one I found later in a different research line, where extending a run’s end date changed results for years that were already over.
Counts of the impossible. Single-day moves beyond 50%, prices under a cent passing a $10 filter, feature importance of exactly zero, or a year of returns that no strategy has ever produced. Several of the leaks in this post would have shown up in a count like that well before they showed up in a result.
Point-in-time by construction. Resolve index membership per date, from the best source you can find, and key vendor fields on when they became knowable, not on the period they describe.
Compare against something real. A copy of an index fund that beats the fund, or two vendors that disagree, is a leak or a data bug until proven otherwise.
A pure-noise null. Run the whole pipeline on random-walk prices, where there’s no signal by construction. If it finds one, it leaks. In one research line, after four leak fixes, the null finally sat on zero, and that was the result I trusted most.
Replication on independent data. When an agent hands you an encouraging statistic, rerun it on data it hasn’t seen before you believe it.

This describes software I run on my own accounts. It is not investment advice and I do not offer investment services.

I let AI agents write most of a live-trading codebase. Here's the gate that made it safe.

Mon, 28 Sep 2026 16:50:00 -0700

Image credit: AI-generated

For the last 21 months I have been the only engineer on TopSet, a system that trades real money. It picks a portfolio with machine-learning models, rebalances on a schedule, and executes through live broker APIs on infrastructure I own end to end. It is my own capital, not anyone else’s.

Agents wrote most of the code. Claude Code, Cursor, Copilot. About 1,600 pull requests in those 21 months, which is more than I could have typed in three times the span.

That leaves you with a choice everyone adopting these tools runs into. You can read every line the agent produces, in which case you have bought yourself a slower version of writing it yourself. Or you can skim, merge, and move fast, in which case you accumulate code that looks right and isn’t. On a system that places orders with real money, the second option ends with trades you didn’t intend.

I took a third option: make something other than my attention the thing that decides what ships. Tests became the gate. Nothing merges or deploys without a green build, and my own reading moved up a level, to design, to test coverage, and to the handful of results I refuse to let change quietly.

That worked. It also failed in ways I did not expect, and the failures are the interesting part, so I’ll spend most of this post there.

What I actually read

“Tests are the gate” is not the same as “I stopped looking.” What changed is where I spend attention.

Most pull requests get a light review. Research code I skim. Changes to the parts that decide what to buy or how to trade it, the model training and the execution paths, I read properly, looking for whether the approach is reasonable rather than whether every line is correct. The tests cover the lines. I cover the idea.

Then there is one rule I do not bend. If a change moves a snapshot, or shifts the result of an end-to-end test, that is a stop. Snapshots and integration results are supposed to be stable, so a diff in one of them means either the change did something I didn’t intend, or it did something I did intend and I now owe an explanation. An agent will happily regenerate a snapshot to make a build go green. That is the single most dangerous thing these tools do, and it is why the snapshot files live in the repo where a diff is visible in review.

For anything involving a calculation I have not verified before, the tests are not enough either. I have the agent write a report script, and there are now 161 of them, that dumps the pipeline to CSV. I open the CSV in a spreadsheet and check the arithmetic by hand: return per trade, slippage, annualized return. Then I spot-check inputs, picking a few prices and going back to the raw data directory to confirm those were the prices the system actually used.

It is slow and it is not clever, and it has caught things nothing else would. A test asserts that a function returns what the author believed it should return. A spreadsheet asks whether the number is true.

What the gate is

Every pull request runs:

lint (ruff) and type checking (mypy)
about 7,500 tests across 331 test modules
an assertion that the test database is empty afterward
a migration rollback-and-reapply, all the way to base and back
terraform validate against both AWS accounts

If any of that fails, nothing merges. The build is the merge gate, and it is not advisory.

Once a week, on a schedule, a heavier set runs:

two end-to-end rebalancing runs against a mock broker
the same integration test 100 times in a row, looking for flakes
nine deterministic model-training snapshot runs, pinned inside Docker
two leakage checks on the training pipeline

Running the same integration test 100 times only makes sense because the test is not the same each time. The mock broker fills orders probabilistically: a 20% chance per 10-second cycle, 70% of those complete and the rest partial at a random size, at random prices. Every iteration produces a different interleaving of fills, partial fills and timing. A hundred runs is a search for race conditions, not a hundred repeats of one path.

What it finds are real concurrency bugs wearing flaky-test costumes. Worker threads leaking between tests. A queued task waking up after teardown had already deleted the row it wanted. A data loader that wasn’t thread-safe and corrupted a shared index under load. Each one showed up first as “that test fails sometimes.”

The weekly set exists because a flaky gate is worse than no gate. If a test fails once every thirty runs, and twenty agent PRs land in a week, that one test goes red most weeks. Have a few like it and something red is always on screen, and you start ignoring red. Hunting flakes is not hygiene here. It is what keeps the gate meaningful.

The tests that matter are about money, not functions

Unit tests are cheap and agents write them happily. They are not what I trust.

What I trust are end-to-end tests that drive a full rebalancing against a mock broker, with the awkward parts turned on. They’re named after the ways money goes wrong:

orders that have to be resubmitted
a rebalancing that resumes from the middle, with partial fills already on the books
a resume that has both buys and sells outstanding
smart orders cancelled mid-flight, on the sell side and then the buy side
deposits and repeating withdrawals arriving while a rebalancing is in progress

These are exactly the control flows a language model gets plausibly wrong. Interrupted work, partial state, money arriving in the middle of an operation: the code for each of these reads fine in isolation. It’s the interaction that bites.

Here is one that made it past review and got caught by the constraint at the bottom of the stack. When the last pending buy of a rebalancing had its amount adjusted for pending withdrawals, the adjusted value could land on exactly zero. The existing guard raised an error on a negative amount, but zero isn’t negative. The affordability check, available_cash >= amount, was 0 >= 0, so it passed. The order was promoted, and the database rejected it: amount > 0 is a check constraint on that table.

The interesting part is what that did to the retry. The integrity error fired inside the same transaction that carried the completing order’s status update, so the rollback discarded that too. The scheduler came back around, found identical inputs, and did it again. Not a self-healing retry: a stuck rebalancing.

The fix took three attempts, and the second one is the useful story. Gating the promotion on amount > 0 looks correct and does nothing, because the code had already assigned the adjusted value to a live ORM object. That marked it dirty, and the next autoflush in the same transaction persisted the zero regardless of whether the order was ever promoted. The condition guarded the decision, not the mutation.

I only know that because the failure was reproduced live against main before any fix was written, and the first fix was re-run against that same repro rather than checked by reading the code. When an agent is writing the patch, “I read it and it looks right” is the single least reliable signal available to you.

Three ways a green build lied to me

This is the part I’d want to read, so here are the three worst cases, all real, all from this codebase.

1. The test asserted a convention that does not exist

Corporate actions have to be applied to historical prices, and stock dividends are the fiddly ones. My broker’s API returns a rate for each. The code treated that rate as a bare fraction: a 5% stock dividend arrives as 0.05.

It isn’t. It’s the ratio of new shares to old. Every one of the 101 stock-dividend records across 43 symbols in my data has a rate of at least 1.0016. Not one is a bare fraction. The convention the code assumed does not occur in the data at all.

The unit test passed, because the test asserted the same wrong convention with a fixture value of rate: 0.05, a number that cannot appear in production. The test and the code were wrong in exactly the same way, which is what you get when both are written in the same sitting from the same misunderstanding.

The effect was that each stock dividend roughly halved the adjusted price, and it compounded across events. One symbol with nine of them came out deflated by about 490×, turning a real price history into a fake ramp of more than a thousand-fold over the period. And because every price stayed between a few cents and a few hundred dollars, a check for implausible price levels would not have flagged it either.

Southern Copper’s real daily prices, adjusted twice by the same code. The only difference is whether the stock-dividend factor is 1 / rate or 1 / (1 + rate).

The fix was a corrected fixture, plus a regression test that runs a nine-event sequence and asserts the cumulative adjustment factor stays sane. The lesson I took: a fixture is a claim about the world, and nobody reviews fixtures.

2. The test passed on NaN

In a research tool, an agent added a second way to select a model configuration and reported the result in the PR: 100% agreement with the existing selector, at every sample size tried.

That number was not real. The tool built its pool from a source that was missing the columns the default scoring version needs, so every score came out NaN. With all scores NaN, the selector fell through to a fixed tie-break order, which of course agrees with itself 100% of the time at every sample size.

The test that was supposed to cover this had been passing on those same NaN scores for as long as it had existed. It never checked anything. Green, meaningless, and quoted in a PR description as a finding.

The follow-up PR made the condition a hard error rather than a silent NaN, pinned the test to a scoring version where the columns exist, and added guards. It also flagged that an older result produced the same way might carry the same artifact. That is the right instinct: when you find a lie, go and check what else it told you.

3. The change did nothing at all

An agent wired a new training target through what looked like the right code path. The first screen ran clean and produced numbers bit-for-bit identical to the baseline. Every metric, to four decimal places.

Identical results are not a pass. They’re a smell. Two different configurations producing byte-equal output usually means one of them didn’t happen.

It hadn’t. The runner always sets a parameter that routes execution to a completely separate function, which builds its own training target and never looked at the new field. Unpickling both runs and diffing the actual selections showed 238 of 238 identical picks for the test year. After the real fix, 0 of 238 were identical, and a regression test now asserts the two modes produce different output.

Nothing failed. The compute was spent. The result would have gone into the research log as “no effect,” which is a conclusion, and it would have been wrong.

The shape of all three

In every case the build was green, and the number was wrong. Tests prove that the code does what the tests say. They say nothing about whether what the tests say is true, whether the code under test ran at all, or whether the data going in is what you think.

That is the agent-era failure mode, and it is worse with agents than without them, for a simple reason: an agent writes the code and the test together, from one reading of the problem. If the reading is wrong, both artifacts are wrong, and they agree. A human pair would at least have had two readings.

What I do about it now, concretely:

Distrust results that are identical, unchanged, or perfect. Check the output, not the exit code.
Treat fixtures as claims. If a fixture value can’t occur in production, that’s a bug in the test.
Make impossible states loud. NaN, missing data, and empty inputs should raise, not flow through and produce a number.
When something is found to be wrong, go back and check what else that thing told you.

The reviewer that isn’t a test

Some mistakes are not code-shaped and no test will ever catch them.

So I added a second kind of review: a deliberately skeptical reviewer persona, run against a finished result rather than a diff, with instructions to argue that the result is not real. I iterated on the persona with Claude until it stopped being agreeable.

It found things I am glad it found. That a configuration had been tuned by repeatedly checking results on data I had promised myself I would only look at once, so my holdout wasn’t a holdout any more. That a cost model was running with zero financing cost while the strategy it priced used nearly four times leverage.

Neither is a bug. Both are the kind of error that invalidates a conclusion, and no test suite in the world was going to tell me.

The gate after the gate

CI stops caring the moment something merges. After that the system runs unattended, on a schedule, against a live broker, and the question changes from “is this code correct” to “is this thing behaving right now.”

Every ERROR the system logs in production becomes a CloudWatch alarm that posts to a Discord channel on my phone. That is the whole alerting stack, and for a one-person operation it is enough: I am not running a rotation, I just need to know.

The loop from there is the same one I use for everything else. Point an agent at the production logs, hand it a local copy of the database from around the failure, ask for a diagnosis and a fix with a test. The logs and the data are usually enough for it to find the problem, and the test it writes is the thing that stops the problem coming back. Several of the bugs in this post arrived that way rather than through CI.

Worth saying plainly: the tests did not catch those. The tests caught the next occurrence.

What stays human

Architecture. Test design. What “done” means. Merge judgment when the tests pass and something still feels off.

And one more thing that turned out to matter: the repo’s own instruction file. Every time an agent reached a wrong conclusion in a way another agent would repeat, I wrote the correction down there. One rule says that any claim of data leakage must name the specific date boundary it violates, because two different agents had asserted leakage from reasoning alone, and both were wrong. That file is now a list of the mistakes this codebase invites. It’s the closest thing I have to institutional memory for a team of one.

What it cost

The per-PR suite takes about ten minutes. That is the tax on every change, and it is the number I would defend hardest: ten minutes is short enough that I never route around it, and long enough to run tests that touch a database and a fake broker.

The machine bill is every one of the 3,000 GitHub Actions minutes my plan includes, plus roughly $20 a month for extra minutes on top. The agents themselves are Claude Pro and Cursor Pro, $20 each. I also paid $40 a month for Copilot until GitHub moved it to per-token billing, which would have raised my bill more than tenfold, so that work moved to Claude Code and Cursor. So the tools that wrote most of this codebase cost tens of dollars a month, like the compute spent checking them, and neither is what the project actually cost.

What it actually cost is fixtures and flakes. Writing a mock broker that fails in realistic ways, maintaining a self-contained dataset so model training can be tested at all, and then the long tail of chasing tests that fail once in thirty runs. None of that is glamorous and all of it is load-bearing. Every hour I skipped there came back later as an hour of not being able to trust a result.

And sometimes the gate itself is the thing that’s broken. The 100-run soak lives in a shell script rather than inline in CI, because the inline version ran under a POSIX shell where {1..100} isn’t brace expansion. It was a literal string. The loop ran exactly once, and reported success, for longer than I’d like to admit.

The tests are not where I over-built. Every gate in this post exists because something broke, or nearly did, and the existing checks let it through. I would add all of them again.

Where I over-built is everything the tests had to cover.

I started TopSet expecting to turn it into a product, so I built it for customers who never existed. A bot lifecycle with eleven kinds of change request, each one a queued, locked, asynchronous workflow object, so that deposits and withdrawals and stops could be requested safely by someone who wasn’t me. Two broker integrations instead of one. A staging environment that mirrors production, behind a VPN. Cross-account backups with compliance-grade retention locks on a database that holds my own trades.

For one person running their own accounts, several of those could have been a script I ran by hand while watching it.

That is the part that compounds, and it is why I am putting it in a post about testing. Every one of those surfaces is something the gate has to cover. The change requests need end-to-end tests for money arriving mid-rebalance. Two brokers means two of everything, including the mocks. The staging environment is a second place for migrations to fail. I did not pay for the product shape once, in the building. I have paid for it every week since, in the size of the harness that has to keep it honest.

If I were starting again for a book of one, I would build the trading and the research properly and leave the product machinery until somebody asked for it.

If you’re rolling this out to a team

I’ve done this alone, with tests I wrote and one person’s judgment. That is not the same problem as a team of fifteen, and I’d be careful about anyone who tells you otherwise. But some of it transfers, and the ordering I’d insist on comes from a team I did run: at a healthcare company, requiring automated tests on every PR is what took active medium and high bugs down 72% and weekly hotfixes from 7 to 1.5. That was before agents. It’s the same first move.

Fix the gate before you raise throughput. Coverage on the paths that lose money or data, and no tolerated flakes. Agents multiply whatever your CI already is.
The author owns the change, whoever typed it. “The agent wrote it” is not a review comment, and it is not a defense in a postmortem.
Review the result, not the diff. For anything that produces a number, ask what would look different if the change had done nothing.
Measure escaped defects and hotfix rate. Not lines, not PR counts. Agents make those metrics meaningless overnight.

The part I would not change: the gate is what let one person move at this speed on a system where mistakes cost real money. The part I underestimated: a gate is a claim about correctness, and claims need checking too.

This describes software I run on my own accounts. It is not investment advice and I do not offer investment services.

Running MosaicML's MPT-7B, a ChatGPT Competitor, on AWS SageMaker

Tue, 18 Jul 2023 08:48:01 -0700

In this blog post, I’m going to take you step-by-step through the process of running MosaicML’s ChatGPT competitor, MPT-7B, on your own AWS SageMaker instance.

MPT-7B and AWS SageMaker. Image credit: MosaicML and Author

Are you excited about the capabilities of ChatGPT, but have concerns about exposing your sensitive data to OpenAI? Luckily, there are alternatives that you can run on your own infrastructure. One such alternative is MosaicML’s MPT-7b, a competitor to ChatGPT, which we’ll explore in this blog post.

Introduction to MosaicML and MPT-7B

MosaicML, recently acquired by Databricks for $1.3 billion, has been making waves in the ML community with their MPT-7B model, a supposed competitor to ChatGPT. Despite its promise, running this model can be daunting due to sparse documentation and its heavy resource requirements. However, one can run MPT-7B on AWS SageMaker in a Jupyter notebook, an environment that is beginner-friendly and highly flexible for rapid iteration. This setup allows you to test the model’s feasibility and hardware requirements before deciding to move into production.

Running MPT-7B on AWS SageMaker

Running MPT-7B in a Jupyter notebook on AWS SageMaker provides several benefits. Not only can you pay just for what you use and turn it off when you’re done, but the ability to easily rerun portions of your code without having to reload the model saves time during iterative development. But do beware! If you forget to stop your notebook instances, the charges can quickly add up.

While this method is relatively convenient, there are some considerations you must take into account. Firstly, loading the model can take up to 20 minutes even on a high-performance GPU, making this process somewhat time-consuming. Also, the cost is a factor to consider, as the running cost is at least $4 per hour. You’ll need to run MPT-7B on at least a p3.2xlarge instance; anything smaller does not seem feasible. If you opt for EC2 instead of SageMaker, you’ll have to ask AWS for permission to use a p3.2xlarge instance.

In the next sections I’ll take you step-by-step through how to run the MPT-7B model in your very own SageMaker jupyter notebook:

Step 1 - Open the SageMaker Console

Fire up the AWS Console and search for SageMaker:

Search for SageMaker. Image credit: Author

Step 2 - Create a notebook instance

From the left-side menu, select Notebook->Notebook instances:

Notebook instances. Image credit: Author

Click the Create notebook instance button:

Create notebook instance. Image credit: Author

Specify an instance name. Choose the instance type m1.p3.2xlarge. Unfortunately, it seems that an instance as powerful as m1.p3.2xlarge is required, or else your instance may run out of memory or take an excessive amount of time to respond to even the simplest questions. However, please note that this instance will cost approximately $4/hr, so it’s important to monitor your usage carefully.

Specify notebook instance details. Image credit: Author

Create a new IAM role:

Create a new role. Image credit: Author

If your test environment doesn’t have any particularly sensitive data in it then you can grant access to Any S3 bucket. Otherwise, you’ll need to be more explicit.

Specify S3 access. Image credit: Author

Click the Create notebook instance button:

Create notebook instance. Image credit: Author

The notebook will then be in a Pending status. This will likely last for about 10 minutes:

Pending notebook instance. Image credit: Author

In the meantime, we’ll download a notebook so that we can upload it after the AWS SageMaker instance has finished provisioning.

Step 3 - Download the notebook

Head on over to the notebook at MPT-7B on AWS SageMaker.ipynb and download it:

The notebook on Google Colab. Image credit: Author

Download the notebook. Image credit: Author

In this notebook you’ll notice two main code blocks. The first block loads the MPT-7B tokenizer and model:

from torch import cuda, bfloat16
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig

device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu'

tokenizer = AutoTokenizer.from_pretrained("mosaicml/mpt-7b-chat",
 trust_remote_code=True)

config={"init_device": "meta"}
model = AutoModelForCausalLM.from_pretrained("mosaicml/mpt-7b-chat",
 trust_remote_code=True,
 config=config,
 torch_dtype=bfloat16)

print(f"device={device}")
print('model loaded')

The tokenizer is used to encode the question sent to the model and decode the response from the model. Additionally, we obtain the device specification for our GPU so that we can configure the model to utilize it later:

import time
from IPython.display import Markdown
import torch
from transformers import StoppingCriteria, StoppingCriteriaList

# mtp-7b is trained to add "<|endoftext|>" at the end of generations
stop_token_ids = [tokenizer.eos_token_id]

# Define custom stopping criteria object.
# Source: https://github.com/pinecone-io/examples/blob/master/generation/llm-field-guide/mpt-7b/mpt-7b-huggingface-langchain.ipynb
class StopOnTokens(StoppingCriteria):
def __call__(self, input_ids: torch.LongTensor,scores: torch.FloatTensor,
 **kwargs) -> bool:
 for stop_id in stop_token_ids:
 if input_ids[0][-1] == stop_id:
 return True
 return False

stopping_criteria = StoppingCriteriaList([StopOnTokens()])

def ask_question(question, max_length=100):
 start_time = time.time()

 # Encode the question
 input_ids = tokenizer.encode(question, return_tensors='pt')

 # Use the GPU
 input_ids = input_ids.to(device)

 # Generate a response
 output = model.generate(
 input_ids,
 max_new_tokens=max_length,
 temperature=0.9,
 stopping_criteria=stopping_criteria
 )

 # Decode the response
 response = tokenizer.decode(output[:, input_ids.shape[-1]:][0],
 skip_special_tokens=True)

 end_time = time.time()
 duration = end_time - start_time

 display(Markdown(response))

 print("Function duration:", duration, "seconds")

Note the use of stopping_critera, which is needed or else the model will just start babbling on, even after it has answered our question.

See model generate parameters if you want to explore the different options.

Now, let’s upload this notebook to SageMaker.

Step 4 - Upload the notebook

Hopefully by this time your SageMaker notebook instance has finished being provisioned. When it has, click the Open Jupyter link:

Open Jupyter. Image credit: Author

Then, click the Upload button in the top-right corner of your screen and select the notebook that you just downloaded:

Upload the notebook. Image credit: Author

Set the kernel to conda_python3:

Set the kernel. Image credit: Author

Step 5 - Run the notebook

Select Cell -> Run All:

Run all cells. Image credit: Author

An hourglass logo will then appear in the browser tab:

Wait for notebook. Image credit: Author

You’ll then need to wait about 10 minutes for the model to be downloaded:

Download model. Image credit: Author

After it runs, you’ll see the answer to the question Explain to me the difference between nuclear fission and fusion:

Explain to me the difference between nuclear fission and fusion. Image credit: Author

Since the model and tokenizer have already been loaded above, you can simply modify the ask_question code block and click the Run button to ask any other questions. This will save you from spending 10 minutes each time you want to test a new question.

What is the capital of France?. Image credit: Author

Step 6 - Stop the notebook

As soon as you have finished testing the model, you’ll want to head back to your list of notebook instances and stop it. If you don’t, $4/hr will add up very quickly 💸

Stop the notebook. Image credit: Author

Performance Comparison

In terms of performance, my preliminary tests suggest that MPT-7B’s results might not be as good as ChatGPT’s. It does a decent job answering questions like What is the capital of France?, Explain to me the difference between nuclear fission and fusion, and Write python code that converts a csv into pdf. But for questions like_What is the capital of Belize?_ it fails pretty horribly:

What is the capital of Belize? Image credit: Author

I am currently collecting more data and will conduct a comprehensive comparative analysis in a follow-up blog post. In that post, I will compare the question and answer performance of MPT-7B, MPT-30B, Falcon-40b, and ChatGPT using actual conversation history.

From Testing to Production

Once you’re ready to transition from testing to production, SageMaker offers an additional benefit - the ability to create an endpoint for your model. With SageMaker, you can auto-scale based on demand to the endpoint, optimizing your resources.

Additional Tips

Be mindful that it’s easy for your process to get forked while running in a Jupyter notebook and run out of memory. If this happens, simply shut down the kernel and run all the commands again.

If you’re curious about running this model on a platform other than AWS, Google Colab Pro is another viable option at $9/month. However, based on our testing, we found that we exhausted the provided credits within just a few hours. 😳

Another challenge you may face is the inability to utilize the Triton optimization on SageMaker due to a CUDA version incompatibility. Unfortunately, AWS’s current P3 instances do not include a recent CUDA version. Therefore, if you wish to utilize the Triton optimization, you will need to create an EC2 container with command line access. However, it’s important to note that you will also require special permission from AWS Support to run an instance with 8 VCPUs. In a future post, I will provide a detailed guide on how to integrate Triton and utilize a more cost-effective GPU cloud provider, such as Lambda Labs.

Final Thoughts

While MosaicML’s MPT-7B offers a viable alternative to OpenAI’s ChatGPT, it presents its own set of challenges. Running the model can be time-consuming, expensive, and the available documentation is lacking. However, the ability to keep the model in-house and protect your data from being exposed to OpenAI can be compelling for certain use cases.

SageMaker offers great convenience for quickly testing the model and provides the flexibility to transition to production when you’re ready. Whether you’re just starting with MPT-7B or have been using it for a while, we hope this guide has provided valuable insights.

Stay tuned for our next blog post, where we’ll delve deeper into the performance comparisons between MPT-7B, MPT-30B, Falcon-40b, and ChatGPT.

See the following links if you are interested in learning more about MPT-7B or its larger variant, MPT-30B.

And remember, whether you’re working with ChatGPT or MPT-7B, the key is to ensure your use case is served without compromising data privacy and cost-effectiveness. Happy tinkering!

Want a Turn-Key Solution for Using MPT or ChatGPT with Your Data?

At MindfulDataAI.com, we offer ChatGPT for businesses. If you’re interested in leveraging ChatGPT, MPT, or other models with your company’s data, please get in touch with us.

My Journey with ChatGPT for Business: Unleashing the Power of AI for Software Engineers

Sat, 08 Jul 2023 07:16:09 -0700

Image credit: D koi on Unsplash

Introduction

As a software engineer, I’m constantly searching for tools and technologies that can revolutionize the way I work. Today, I want to share a prototype app: ChatGPT for Business. I believe if we follow through on building this, it has the potential to transform our industry. Join me as I take you through my experiences, challenges, and the immense possibilities that lie ahead.

From Crypto Trading to AI-Assisted Productivity

The story begins with me leaving my Director of Engineering job at a fast-growing startup that was about to be acquired. I stumbled into the world of crypto trading after attending a web3 event in early 2022, where I heard about arbitrage opportunities. Eager to explore further, I began experimenting with free bots at Pionex and discovered the passive income potential of trading bots. That’s when everything changed.

Driven by curiosity, I reverse-engineered a promising bot, conducted extensive backtesting using historical price data from Binance, and optimized its parameters using genetic mutation techniques. Deploying it on AWS, I witnessed consistent daily returns, achieving a remarkable 18% annual return on a sizable investment. But as the market landscape shifted, I realized that luck played a significant role in my success, and a long-term buy-and-hold strategy would yield better results. When the SEC’s lawsuit against Binance and Coinbase loomed, I secured my crypto in wallets and closed my trading operations.

Discovering the Power of ChatGPT

During this transformative period, I found myself increasingly relying on ChatGPT, a powerful generative AI model. What began as a tool for trivial questions quickly evolved into something much more. ChatGPT gradually replaced around 80% of my Google searches, becoming indispensable for coding technical analysis algorithms, troubleshooting Docker configurations, proofreading emails, and even generating pitch deck outlines. I realized that ChatGPT had become my go-to resource, surpassing Google searches in many aspects. However, it was missing the capability to search my private data and access any information released after 2021.

The Spark of Innovation

One day, over coffee, a friend shared an interesting challenge: scraping information from multiple mom-and-pop websites to create a comprehensive activities-for-kids marketplace. It was at that moment that I had a revelation—what if ChatGPT could help? Instead of writing custom web scrapers for each site, could ChatGPT extract the desired information? Intrigued, I stumbled upon an informative post titled “How to Train an AI Chatbot With Custom Knowledge Base Using ChatGPT API”. Inspired by its possibilities, I pointed ChatGPT to a Boys and Girls Club website, asking questions about activities in Seattle, their schedules, and costs. To my delight, the results were remarkably accurate, and the cost of using OpenAI’s API was surprisingly affordable.

Unlocking the Power of ChatGPT for Business

Fueled by this success, I couldn’t help but wonder how I could further enhance my productivity using ChatGPT. To my surprise, I discovered that very few business products allowed users to leverage ChatGPT with their own data. Existing solutions fell short in terms of integrations, customization, and scalability. MyAskAI catered to individuals with one-off files and only allowed two questions per message thread. AiSera seemed more focused on building chatbots with custom data. CustomGPT lacked the necessary integrations with data sources like Google Drive, Salesforce, and Confluence. Gerev.ai, although open-source and appealing, failed to index a 5-page PDF on a brand-new MacBook Pro. Klu had an impressive turnkey integration with sources like Google Drive, but its “Beta AI” spun endlessly when asked a basic question about a single 5-page PDF file.

This realization sparked a vision in me—I wanted to build a product specifically designed to empower all businesses, not just the ones with the funds to train new AI models, with ChatGPT’s capabilities. Excited by the possibilities, I reached out to my friend Arty, a talented engineer, and we began brainstorming ideas together.

Seeking Validation and Refining the Vision

To refine our concept and gain support, we sought guidance from a local venture capitalist (VC). We were told that our idea might be too broad and that we should instead focus on a specific industry. However, we remained unconvinced that there isn’t a place for a general-purpose ChatGPT tool for businesses. Therefore, we decided to rely on data to guide us in the right direction. Undeterred, we developed a playbook, crafted surveys, optimized keywords for SEO and advertising, and created compelling website mock-ups. But we knew that true validation would come from our fellow software engineers—the very people who could benefit from ChatGPT for Business—YOU!

Imagine, if you will…

A ChatGPT-powered teammate, seamlessly integrated with your data sources and the web, providing real-time insights, securely linked to source documents, and offering turnkey integrations with your company’s tools and systems. We envision automated document synchronization, real-time web searches, single sign-on (SSO) integration, access controls, conversation sharing, onboarding and training, and robust security and compliance measures.

Imagine being able get the status of a Jira ticket in real-time:

Image credit: Author

Or, what about keeping tabs on the competition?

Image credit: Author

Or, even synthesizing reports from raw data. Sharing and explaining insights (like a goalie, answering questions before you need to!). The more documents you share with the agent, the more time it will save for you as you could even let this AI agent handle basic questions on your behalf!

Image credit: Ice Hockey Dancing by NHL

The possibilities are endless.

Join the Journey

And now, dear reader, we invite you to embark on this incredible journey with us. We think we’re on to something, but we want your thoughts! Your insights and feedback are essential as we shape the future of ChatGPT for Business. Together, we can unleash the potential of ChatGPT, transforming the way we work, and empowering ourselves and our teams with a tool that will revolutionize communication. Please take a moment to share your thoughts by participating in our quick survey. Additionally, you can also register for the beta release.

Conclusion

Our journey with ChatGPT for Business has been one of passion, innovation, and relentless pursuit. We have witnessed firsthand the immense potential of AI in our field, and we’re determined to harness that power to create a tool that will revolutionize software engineering. Thank you for reading, and we eagerly await your input as we shape the future of ChatGPT for Business together.

Run a Free VPN Server on AWS (2022 Edition)

Tue, 15 Nov 2022 10:48:50 -0800

This tutorial will take you through the steps of setting up an EC2 instance running OpenVPN server. It will then cover how to grant and revoke access to your VPN server.

Image credit: Privecstasy on Unsplash

AWS has an awesome firewall built into its core services, which can easily be used to make sure that only certain ports are open to the outside world. One extra step that we can take is to run a VPN server that serves as the gateway to our protected AWS resource, e.g. EC2, RDS, etc… We can then shutdown direct access to our AWS resources just by revoking access via our VPN server. This is very useful if you need to revoke access for a former employee.

AWS now offers a managed VPN Service, but this service costs at least $72 a month and is even more expensive when your VPN serves a lot of traffic. A lot of smaller organizations don’t need all the features of the managed service and instead can run their own VPN server for just the cost of an EC2 instance. You can even use the free tier so that you don’t have to pay anything for the EC2 instance for the 1st year.

Step 1 — Create the VPN Security Group

Overview: security groups allow your servers to communicate with each other in a private cloud while exposing specific ports to the world. We are going to create a security group to allow VPN access to our VPN server. We will assume that all your other AWS resources are members of the default security group and that the default security group does not allow access from the outside world.

From the EC2 dashboard, click Security Groups:

Image credit: Author

Click Create security group:

Image credit: Author

Enter a name and description of vpn and specify the following inbound rules on ports 22, 443, 943, and 1194. Note: the protocol for port 1194 is UDP.

Image credit: Author

Note: if the IP addresses that your team uses are static then you can add yet another layer of security by specifying an IP address range in the Source of your rules. However, you’ll want to leave the Source open to anywhere if you want your team to be able to connect from any IP as they may be working from a hotel, home, cafe, etc…

Step 2 — Create the EC2 Instance

Return to the EC2 Dashboard and then click Launch instances:

Image credit: Author

Select Ubuntu (you can of course select almost any other OS that runs OpenVPN, but this tutorial is tailored to Ubuntu)

Image credit: Author

Select t2.micro:

Image credit: Author

Note: you can use a nano instance instead of a micro instance, but nano instances are not eligible for the free tier.

In the Network settings section, select your default VPC and disable the auto-assign public IP option. Then, select both your default security group and the security group that you created above for the VPN.

Image credit: Author

Click Launch instance

Step 3 — Disable Source/Destination Check

From the list of instances, select the VPN instance and then Actions -> Networking -> Change source/destination check from the drop down menu.

Image credit: Author

Select Stop and click Save. This is needed as otherwise, your VPN server will not be able to connect to your other AWS resources.

Image credit: Author

Step 4 — Create an Elastic IP Address

Overview: when an EC2 instance is stopped and restarted, the Public IP address changes. We want the IP address of our VPN server to remain static so we’ll use an Elastic IP Address.

From the E2C Dashboard, select Elastic IPs:

Image credit: Author

Click Allocate Elastic IP address:

Image credit: Author

Make a note of your new Elastic IP address as this will be the Public IP Address of your VPN server. We will refer to this address later as PUBLIC-IP-OF-VPN-SERVER.

Select the IP address you just created and click Associate Elastic IP address:

Image credit: Author

Then, select the Elastic IP and click Associate address from the drop down menu.

Select the EC2 instance you created above and click Associate:

Image credit: Author

Step 5— Install and Configure the OpenVPN Server

SSH into your VPN server:

$ ssh ubuntu@PUBLIC-IP-OF-VPN-SERVER

Download our helper scripts and set up a default config:

$ git clone https://github.com/redgeoff/openvpn-server-vagrant
$ cd openvpn-server-vagrant
$ cp config-default.sh config.sh

Edit config.sh and enter in your configuration. Note: PUBLIC_IP should be equal to the Elastic IP Address that you created above.

$ nano config.sh

Switch to root:

$ sudo su -

You’ll now update Ubuntu. Note: you will be prompted several times and when you do, just press the Enter key.

$ /home/ubuntu/openvpn-server-vagrant/ubuntu.sh

Now, install OpenVPN. Note: you will be prompted several times and when you do, just press the Enter key.

$ /home/ubuntu/openvpn-server-vagrant/openvpn.sh

At this point, the OpenVPN server is running.

Step 6 — Add the Route

Routes must be added to the server so that your team’s clients know which traffic to route to the VPN server.

You can determine the proper subnet by returning to your list of EC2 instances, clicking on a target instance and identifying the Private IP.

Image credit: Author

Your network will be the first 2 parts of the Private IP appended with zeros, e.g. 172.31.0.0

On the VPN server edit /etc/openvpn/server/server.conf and add something like the following:

push "route 172.31.0.0 255.255.0.0"

Then, restart the VPN server with:

$ systemctl restart openvpn-server@server.service

Step 7 — Grant Access to Your VPN

Note: We assume that you are still SSH’d into the VPN and logged in as root.

Run the following command and be sure to replace client below with a unique name for your user/client.

$ /home/ubuntu/openvpn-server-vagrant/add-client.sh client

You’ll then find a configuration file at

~/client-configs/files/client-name.ovpn

You will want to provide this file to the individual on your team who will be connecting to your VPN. SCP is handy for downloading this .ovpn file from your VPN server.

Your team can use one of various VPN clients such as Tunnelblick (OS X) and OpenVPN (Linux, iOS, Android and Windows). After installing one of these clients, they should be able to set up the VPN config just by double clicking on the .ovpn file.

Note: once connected to the VPN, your users will want to use the Private IPs of your AWS resources. You may want to use nslookup to lookup the IP address from the AWS custom domain names provided by AWS. You’ll probably want to use Route 53 to create subdomain records that route to the Private IPs.

Step 8 — Revoke Access to Your VPN

Note: We assume that you are still SSH’d into the VPN and logged in as root.

Run the following command and be sure to replace client below with a unique name for your user/client.

$ /home/ubuntu/openvpn-server-vagrant/revoke-full.sh client

Troubleshooting

If your VPN client reports a TLS handshake failed error then this is most likely because your VPN security group (Step 1) is incorrect. Make sure that you have the correct ports and protocols specified — a common problem is not specifying UDP for port 1194.

We Migrated From AWS Lambda to ECS, but Hope to Eventually Migrate Back

Tue, 17 Aug 2021 07:16:09 -0700

The pitfalls of Lambda that made us switch away

Image credit: Attentie Attentie on Unsplash

Author’s note: The views expressed in this post are my own and not necessarily those of my employer.

This is a story about how we migrated from Lambda to ECS because of a 2% error rate and limited concurrency controls. And, about how AWS has more recently fixed their error-rate issues and how enhancements to concurrency controls could win us back to Lambda. When I first started at Knock in 2018, we were running our entire stack on AWS EC2 instances that we were manually managing.

As a result, it was painful to scale and maintain our infrastructure, and we were wasting money as we were unable to distribute our resource consumption. We were considering a managed container service like AWS Elastic Container Service (ECS) and AWS’s most recent serverless option, Lambda.

AWS claimed that Lambda was a reliable, feature-rich environment that abstracts away any notion of having to manage a server, and we were excited about the prospect of just being able to focus on the code.

Unfortunately, after a year of using Lambda in production, we observed Lambda was a relatively unreliable technology and when it was used to deliver customer-facing features, a 2% error rate in the AWS layer resulted in a very poor user experience. So, we decided to migrate some of our customer-facing Lambdas to ECS.

Before we dive into the details, I’d like to clarify that this post is not a hate letter to AWS about Lambda.

As you will read later, in 2020, AWS made improvements to Lambda networking, which has almost completely eroded the error rates we were seeing.

This leaves just the concern of concurrency, which is something that can be mitigated in certain use cases, but not all.

Because there is one shared concurrency pool for all your Lambdas, a burst of activity in any of your Lambdas can easily lead to significant throttling of mission-critical functionality, which results in an unacceptable user experience.

Overall, I’m hopeful that AWS will eventually enhance the concurrency model for Lambda and remain committed to the promise of a world where engineers spend less time managing servers.

What Were We Running On Lambda?

Let’s start by describing the types of services we had running on Lambda, as after all, we can sometimes tolerate latency and failures in certain use cases.

The first class of services comprises a backend of RESTful APIs fronted by API Gateway and backed by a Lambda function for each route.

We had both Node.js (Typescript) and Python code running in this layer. Although the programming language choices here have little impact on what we discuss in this post, they do impact cost and overall Lambda concurrency and latency.

The main callout here is that most of these APIs were being called by frontends synchronously and thus, significant latency and failure is unacceptable as it manifests as downtime to the user.

The second class of services mostly consists of background tasks that read messages from SQS queues and then perform IO operations on RDS instances and S3 buckets.

In our case, nothing in this pipeline is too time-sensitive, so if these background tasks fail occasionally, a retry mechanism is enough to keep our system healthy and everyone happy.

Up to 2% Error Rates in Lambda From 2019–2020

It wasn’t until we launched RESTful APIs in production that we began to notice that transient errors in the AWS layer were common for AWS Lambda. In particular, we started noticing errors in CloudWatch where a Lambda would timeout before it even ran any of our application-layer code. We knew the application code was not running as the very first thing our application code does is log the incoming request and these log statements from our application layer were missing. In other words, there was nothing that we could tune to prevent this from happening as the issue was occurring in the AWS layers. For example:

START RequestId: 56a12bf8–43eb-4b6c-8a20-c73fa2152c6b Version: $LATEST

[Missing entries from application logs]

END RequestId: 56a12bf8–43eb-4b6c-8a20-c73fa2152c6b

REPORT RequestId: 56a12bf8–43eb-4b6c-8a20-c73fa2152c6b Duration: 30019.42 ms Billed Duration: 30000 ms Memory Size: 1024 MB Max Memory Used: 77 MB

2019–08–22T20:10:31.270Z 56a12bf8–43eb-4b6c-8a20-c73fa2152c6b Task timed out after 30.02 seconds

On August 28th, 2019, We reached out to AWS via a support ticket and they confirmed that a 1–2% failure rate was somewhat normal:

Just to summarize the conversation, we talked about a 1%-2% failure rate not being outside of the realm of possibility in spikes, and the recommended implementation is backoff and retry on the client side.

AWS recommended a retry mechanism, but in practice, the only way to detect these problems was by setting a more aggressive timeout and then retrying.

Some of our legacy code in the critical code path can take several seconds to run during heavy times of load and Lambda cold starts used to be fairly significant.

So, let’s say that we implemented a five-second timeout and then retried at this point. This would result in the user having to wait at least five seconds, which is unacceptable when it happens 2% of the time!

Moreover, our backend is distributed, and therefore operations are not always atomic, meaning that there would be an increased possibility of duplicate requests when aborting the Lambda invocation and retrying the request.

This was a major deal-breaker for us, so we decided to start migrating away from Lambda at the end of 2019.

Shared Concurrency Leads To Throttled Execution and Frustrated Users

One could argue that the concurrency control capabilities of Lambda are severely lacking as there is a single per-region cap on Lambda currency per AWS account.

This means that any bursting Lambda activity can cause customer-facing Lambdas to be throttled. Here, I’m defining customer-facing Lambdas as those fielding synchronous requests from our users via a UI.

In other words, Lambda-powered background tasks can starve these customer-facing Lambdas and can essentially result in your app becoming unresponsive. This manifests as an outage to your users and is completely unacceptable when it occurs frequently.

As illustrated in the next set of graphs, you can see that spikes in customer-facing errors roughly correlate to the spikes in our account’s Lambda concurrency:

Errors from hitting max concurrency. Image credit: Author

This cannot be easily prevented as there is a limit of 1,000 concurrent Lambdas on our account, and the only way to bound Lambda executions is by reserving part of this pool of 1,000. In other words, each time you reserve concurrency for a Lambda, you forever take away potential concurrency for your other Lambdas, resulting in an awkward juggling act where you have some Lambdas that are statically bounded and others that are not.

If you are familiar with AWS Elastic Container Service (ECS) or Elastic Kubernetes Service (EKS), you’ll know that it is fairly trivial to control the number of containers (worker pool) by scaling up or down the number of tasks running for a service. In other words, you have full control over how you allocate your resources and aren’t locked into a primitive per-function concurrency model — like you are with Lambda. In a later section in this post, we’ll investigate how I think AWS could improve its concurrency controls to facilitate better resource sharing.

Given Lambda’s basic concurrency controls, is there anything we can do to prevent throttling our user-facing Lambdas? Yes, you can:

temporarily constrain the concurrency of more Lambdas with reserved concurrency
file a request with AWS support to bump up your concurrency limit from 1,000
increase the batch size on your SQS reads to a maximum of 10. This means that it takes fewer Lambdas to service the same SQS traffic. Let’s just hope you structured your code to work this way before your Lambdas were throttled
set the reserved or provisioned concurrency for user-facing Lambdas. This is fine if you only have a few Lambdas, but if you have one for each RESTful API route, you could have hundreds of Lambdas and you’ll quickly get into a situation where you actually lock up all of your concurrency. For example, imagine you have just 10 Lambdas/routes and set the reserved concurrency at 20 for each of these Lambdas, resulting in a total reserved concurrency of 20 x 10 = 200 (recall you have a max of 1,000 for your entire AWS account). That number can quickly add up and what’s more is that each of these Lambdas is now bound to a max concurrency of 20, meaning that your user-facing Lambdas aren’t going to scale as your traffic increases, which is a big reason why you even use Lambda in the first place.
stand up a SQS queue in front of API endpoints for things that are resource-intensive and/or can be processed asynchronously, e.g., email sending and report generation. This allows the customer-facing Lambdas to complete their execution faster as the resource-intensive work is deferred. Although this is typically a good pattern as it reduces latency to your user, it’s not always enough to keep your Lambda concurrency low enough to prevent throttling and it may just increase the concurrency among your background tasks if you are using Lambda to run your background tasks.
Move your user-facing Lambdas to a different region so that you have access to a different concurrency pool of 1,000. This can greatly increase the complexity of your deployments and it may even mean that other services like Cognito, DynamoDB, SQS, etc., would need to be moved to another region.

A lot of these mitigations are good things to consider when using Lambda, but they don’t solve the root problem of very limited concurrency controls leading to throttling. If you use these mitigations, you’re basically just incurring tech debt that you’ll be on the hook for later.

Poor error rates and unavoidable throttling? These were both deal-breakers, so we had no choice but to migrate away from Lambda.

Refactoring Our Code So That It Can Run On Both Lambda And ECS

Now that we had decided to migrate away from Lambda, we wondered if there was a way to make a minimal set of changes to our code so that it could also run in a Docker environment.

We chose AWS’s Elastic Container Service (ECS) to run our containers as it is relatively easy to manage and something that our DevOps team had the bandwidth to support. Fortunately, we noticed that the Lambda handlers we already implemented utilize a nice abstraction, which could be used to translate from a Docker-supported framework to a Lambda one and vice-versa. All we needed to do was create a wrapper that would:

translate the ExpressJS request payload to the event payload needed by the Lambda handler
translate the Lambda response to an ExpressJS response
use a wrapper that would allow us to reuse all our existing Lambda handler code and run it on top of the ExpressJS framework! We chose to go with ExpressJS to build the RESTful API as we were using Node.js and ExpressJS is a battle-tested framework. The following implementation is in JS but the pattern will work with any web framework and any programming language. (Our actual implementation is in TypeScript.)

Let’s take a look at some code to see how we did this. Consider a simplified version of a Lambda handler that implements a POST /lead/:leadId route, which accepts a JSON body of { name, email }:

import leadService from './lead-service';

// Define a handler for `PUT /lead/:leadId` that accepts the request body
// payload `{ name, email }`
const handler = async (event) => {
 // Parse the request parameters
 const { name, email } = JSON.parse(event.body);
 const { leadId } = event.pathParameters;

 // Assume this is the service-layer code that actually performs the
 // updating of the lead
 await leadService.update(leadId, name, email);

 // Return a 200 success code and the full lead
 return {
 statusCode: 200,
 body: JSON.stringify({
 id: leadId,
 name,
 email
 })
 };
}

export default handler;

We created a lambdaToExpress function that translates from ExpressJS to Lambda and back again, using the following code:

// Convert from a Lambda to an ExpressJS handler
const lambdaToExpress = async (handler) => {
 return async (req, res) => {
 // Repackage the express parameters as an event object for the Lambda
 const event = {
 body: JSON.stringify(req.body),
 headers: req.headers,
 queryStringParameters: req.query,
 pathParameters: req.params
 };

 const handlerRes = await handler(event);

 // Instruct express to use the response from the Lambda 
 res.status(handlerRes.statusCode).json(JSON.parse(handlerRes.body))
 };
}

export default lambdaToExpress;

Then, we registered our ExpressJS routes using our new wrapper. Here is its code:

import express from 'express';
import cors from 'cors';
import handler from './lambda';
import lambdaToExpress from './lambda-to-express';

const PORT = 3000;
const app = express();

app.use(cors());
app.use(express.json()); // Parse JSON bodies

app.put('/lead/:leadId', lambdaToExpress(handler));

app.listen(PORT);

With just a few lines of code, we enhanced our single codebase to support both Lambda and ExpressJS. One thing to note is that both the lambdaToExpress and handler functions are performing a JSON.stringify() and JSON.parse(), which adds a tiny bit of extra overhead. That being said, these operations are very quick and unnoticeable in the grand scheme of things.

The code was all set, but what about anything provided by AWS’s API Gateway?

As part of our API is external-facing, we needed to support rate-limiting. To achieve this, we incorporated node-rate-limiter-flexible (backed by ElasticCache) to implement rate-limiting. And, we simply added this rate-limiting check to our lambdaToExpress wrapper to implement this in a single place.

We were using the Serverless Framework to deploy our Lambdas, and you could probably use this framework to deploy code to ECS, but we find Terraform to be a slightly more versatile tool for configuring non-serverless infrastructure and were already using Terraform to deploy other microservices.

So, we created Terraform Modules alongside our serverless.yml configuration and deployed them with Jenkinsfiles.

And now, we were running on ECS!

We have been happily running a portion of our RESTful API backend on AWS Lambda for over a year and have found that we have reduced our compute costs by 90%! In a future post, I’ll dive into a cost comparison between Lambda and ECS.

Lambda Reduced Its Error Rate From 2% To 0.0000625%

If you recall from earlier, in August 2019, we found that AWS Lambda could suffer from an error rate of up to 2%!

In September 2019, AWS released a blob post touting improvements to networking for Lambda that would reduce latency incurred with cold starts. We were very excited about this news as it meant that AWS had acknowledged the problem and was actively working on a fix.

We had migrated some of our backends from Lambda to ECS but had a lot of remaining functionality on Lambda.

In 2021, we performed another analysis and found that the latest version of Lambda was resulting in an error rate of only 0.0000625%. This is a significant improvement from 2%!

Here is the analysis we did for our auth-service, a RESTful API responsible for authentication and authorization functionality in our system. In a seven-day period, the auth-service had about 1.6 million invocations and only one resulted in a timeout.

auth-service invocations. Image credit: Author

With error rates as low as this, our error rate concern with user-facing Lambdas has been eliminated! But what about the lingering concern with concurrency controls?

How Could AWS Improve Its Concurrency Controls?

If you recall, we ended up performing many of the concurrency mitigations mentioned above and moved a number of our Lambdas to ECS.

Afterward, we were left with a relatively comfortable set of about 200 Lambdas, and as a result, bursting concurrency is now less of a concern.

That being said, we now avoid using Lambda for any new user-facing services. And, we generally avoid using it for substantial background jobs as we are worried that excessive use will result in throttling our few remaining user-facing Lambdas.

Despite this current strategy, I’m hoping the day will come when AWS implements a more feature-rich implementation of Lambda concurrency controls that better mimics that of an ECS environment. Here are some enhancements that would make a huge difference:

Max Concurrency

Let’s define the Max Concurrency of a Lambda as a setting for the max concurrent invocations of a specific Lambda. This value is not the same as Reserved Concurrency in that the value of Max Concurrency wouldn’t subtract from the overall concurrency available for the AWS account.

Here’s an example: assume Lambda A has a Max Concurrency of 100 and your account has a global max concurrency of 1,000 (the AWS default).

Lambda A would not be throttled until there are over 100 instances of Lambda A running concurrently. Other Lambdas, e.g., Lambda B/C, can still consume up to 1,000 instances, if Lambda A has 0 running instances.

This in essence means that no matter what, Lambda A cannot consume more than 100 instances of the total 1,000, but doesn’t require us to set aside 100 instances from our total 1,000 for Lambda A.

This setting is somewhat akin to having an ECS service that can burst to an upper bound of max tasks but doesn’t require us to reserve a portion of our cluster, even if that service is not running all the instances.

Group Max Concurrency

We consider a service to be a group of related Lambdas. For example, consider 10 Lambdas that comprise 10 RESTful endpoints of a service. Or, 10 Lambdas that define the workers of an ETL pipeline. Often, we’d like to define an upper bound on concurrency for a group of Lambdas, e.g., we don’t want our ETL pipeline to consume more than 100 out of our total of 1,000 current Lambda invocations.

A Group Max Concurrency could effectively be accomplished by first allowing you to tag Lambdas to be part of a group. Second, you could then set the Group Max Concurrency for that group. Or AWS could even reuse the concept of a “stack” and have a Stack Max Concurrency.

This Group Max Concurrency is similar to how you may have an ECS service where each container is performing multiple (a group of) operations, e.g., a web server fielding requests for multiple RESTful API endpoints.

Ability to increase account-wide Max Concurrency without contacting AWS support

Some will point out that you can always increase your total Lambda concurrency just by contacting AWS support.

In fact, we actually did this to bump our Full account concurrency from 1,000 to 2,000. AWS support was pretty amenable to this, but it took a couple of days for them to make the change and we needed to provide justification.

All the while, our user-facing Lambdas were throttled, and our users were upset. I’ve even heard of companies being able to get AWS to increase their max concurrency to hundreds of thousands of Lambdas.

That’s great, but why is special approval even needed? Isn’t the fact that a company is willing to pay for the increased usage enough?

To compare with ECS, this max-concurrency-per-account setting is similar to how you can easily customize the number of nodes in an ECS cluster, which effectively defines a max number of ECS tasks that can be run concurrently.

And of course, ECS has autoscaling capabilities that allow you to scale your cluster based on your own needs. None of this scaling requires any special approval from AWS.

If AWS were able to implement these controls, it would bring AWS Lambda’s concurrency controls to parity with those of ECS and EKS, and I’d once again advocate for using Lambda in many more areas of our system.

When Does Lambda Shine?

Even with its limited concurrency controls, Lambda shines in the following areas:

For background tasks, e.g., ones that read off SQS. Sure, you can stand up something like celery or bull to get something similar, but wiring up a Lambda to an SQS queue or one of the many other AWS triggers, is such a beautifully simple thing. Note: Lambda has a max timeout of 15 minutes, so it is only suitable for running relatively quick tasks.
When you have other Lambdas that don’t burst to consume all of the available concurrency, such as when you have a relatively small amount of Lambda activity.
When you wish to optimize autoscaling so that you only pay for what you use. Lambda automatically scales up and down at the single invocation layer so that you can save a ton of money during low traffic periods
As a place to run your code without the need to manage infrastructure

Conclusion

The promise of serverless — where developers don’t need to manage infrastructure — is very compelling, and I don’t want to let this dream die.

AWS has fixed its error-rate problem with Lambdas, but Lambda is still lacking better concurrency controls.

Until these concurrency controls are enhanced, you’ll probably want to use ECS or EKS for your user-facing services, if your Lambda usage is significant and will lead to throttling.

I suspect that AWS will continue to evolve the Lambda technology to the point where it becomes a strong alternative to ECS and when it does, I’ll once again tout the virtues of using Lambda everywhere.

Thanks for reading.

What are MSON Template Queries?

Mon, 08 Feb 2021 00:00:00 +0000

Template Queries are dynamic templates constructed with MongoDB-style operators. They allow you to customize MSON components with less code and are very extensible.

OK, but what the heck is MSON?

MSON is a low-code way to create apps from JSON. The ultimate goal of MSON is to allow anyone to develop software visually. You can also use pieces of MSON to turbo charge your existing apps.

A simple form component used to collect a name and email address could look like:

{
 name: 'MyForm',
 component: 'Form',
 fields: [
 { name: 'name', component: 'TextField', label: 'Name' },
 { name: 'email', component: 'EmailField', label: 'Email' }
 ]
}

You can read more about MSON’s language principles here.

What are MongoDB operators?

MongoDB uses an extensive query language that is written in JSON. Because this language is so powerful, a number of projects, that work independently from MongoDB, have adopted support for the query language:

In-memory libraries for JS, like mingo and sift
Operators in sequelize ORM
Other databases, like CouchDB

This query language also contains a construct for pipeline aggregations, which can be used to execute operations like $add, $multiply, $filter, $map, etc… For example:

{
 $add: [ 1, 2 ]
}

A basic MSON Template Query

Now that you have that background, let’s take a look at an example where we extract the year from a user-provided date. Upon selecting a date with the date picker, you’ll see that the Year field is populated:

Let’s step through this example. First, we construct a form with date and year fields:

{
 component: "Form",
 fields: [
 {
 name: "date",
 component: "DateField",
 label: "Date"
 },
 {
 name: "year",
 component: "IntegerField",
 label: "Year"
 }
 ]
}

Second, we listen for changes to the date value, extract the year and then set the value of the year field:

{
 component: "Form",
 fields: ...,
 listeners: [
 {
 event: "fields.date.value",
 actions: [
 {
 component: "Set",
 name: "fields.year.value",
 value: {
 $year: {
 $toDate: "{{fields.date.value}}"
 }
 }
 }
 ]
 }
 ]
}

The $toDate operator is used to convert the date value, which is in milliseconds, to a date object. The $year operator is used to extract the year portion from the date. The Set action is native to MSON and is used to set the value of the year field.

Pretty cool? Let’s go a bit deeper…

How can we make this reusable?

You can create custom actions, which can then be reused. Let’s assume that we want to create an app.GetYear action that retrieves the year from a DateField value:

compiler.registerComponent("app.GetYear", {
 // Extend Set, a core MSON component, that sets a
 // property on another component
 component: "Set",

 // Define the inputs to the custom action
 schema: {
 component: "Form",
 fields: [
 {
 name: "date",
 component: "DateField",
 required: true
 }
 ]
 },

 // Omitting the "name" here allows the value
 // to be passed to the next action via
 // "{{arguments}}"
 //
 // name: "",

 // Use the template query to extract the year
 // from the date
 value: {
 $year: {
 $toDate: "{{date}}"
 }
 }
});

We can then use this custom action in a chain of actions:

const form = compiler.newComponent({
 component: "Form",
 fields: ...,
 listeners: [
 {
 event: "fields.date.value",
 actions: [
 // Extract the year and pass it to the next action
 {
 component: "app.GetYear",
 date: "{{fields.date.value}}"
 },

 // Use the extracted year to set the year field
 {
 component: "Set",
 name: "fields.year.value",

 // {{arguments}} is the output of app.GetYear
 value: "{{arguments}}"
 }
 ]
 }
 ]
});

Here is the completed example:

Another example: a conditional survey question

Let’s assume that we have a survey question and we want the user to be able to enter a free response when they select Other:

We define the form and hide the covidOther text field by default:

{
 component: "Form",
 fields: [
 {
 name: "covid",
 component: "SelectField",
 label: "My government's response to COVID-19 was...",
 fullWidth: true,
 options: [
 { label: "Good", value: "good" },
 { label: "Bad", value: "bad" },
 { label: "Other", value: "other" }
 ]
 },
 {
 name: "covidOther",
 component: "TextField",
 label: "Please explain",
 multiline: true,
 hidden: true
 }
 ]
}

If the user selects Other, we toggle the hidden boolean property of the covidOther field. The $ne operator is shorthand for not equal.

listeners: [
 {
 event: "fields.covid.value",
 actions: [
 {
 component: "Set",
 name: "fields.covidOther.hidden",
 value: {
 $ne: ["{{fields.covid.value}}", "other"]
 }
 }
 ]
 }
]

Why doesn’t MSON support custom JS in templates?

By now, you can probably see the power of Template Queries, but you may be wondering why MSON doesn’t support custom JavaScript. To support custom JS in template parameters, it would require breaking two of MSON’s core design principles:

Compilation by instantiation - Components are compiled into JS objects by simply instantiating a JS object and setting the props dynamically. This method of compilation allows us to avoid a transpilation step and makes it much easier to dynamically modify components.
Serialization & deserialization without eval() - Components can be serialized using JSON.stringfy() and stored practically anywhere. Moreover, components can be deserialized by dynamically compiling the result of JSON.parse(). As a result, raw JS (in a string), including JS template literals, are not supported as deserializing such JS would require the use of eval() or new Function(), which would expose a XSS vulnerability and add significant performance issues.

You can read more about this reasoning here.

Wrap It Up!

Template Queries add a ton of capability to MSON by providing another powerful way of customizing your MSON components. And, they can be particularly useful in minimizing the code it takes to conditionally chain a series of actions.

If you liked this post, please give it a like or two. Happy building!

How to autogenerate forms in React and Material-UI with MSON

Mon, 22 Oct 2018 00:00:00 +0000

Implementing great forms can be a real time-waster. With just a few lines of JSON, you can use MSON to generate forms that perform real-time validation and have a consistent layout. And, MSON comes with a bunch of cool stuff like date pickers, masked fields and field collections.

Disclaimer: this post is geared towards those wishing to use Material-UI with React. Future versions of MSON will support other rendering layers.

What the heck is MSON?

MSON is a declarative language that has the capabilities of an object-oriented language. The MSON compiler allows you to generate apps from JSON. The ultimate goal of MSON is to allow anyone to develop software visually. You can also use pieces of MSON to turbo charge your form development.

A Basic Form Generated with MSON

Simply declare your form in JSON. Then let the MSON compiler and UI-rendering layer autogenerate your UI:

Did you try submitting the form without filling in any values? Did you notice how real-time validation is automatically baked in?

Now, let’s take a closer look at what is happening. The first block of code contains a JSON definition that describes the fields in the form:

const definition = {
 component: "Form",
 fields: [
 {
 name: "heading",
 component: "Text",
 text: "# Form using [MSON](https://github.com/redgeoff/mson)"
 },
 {
 name: "fullName",
 component: "PersonFullNameField",
 required: true
 },
 {
 name: "birthday",
 component: "DateField",
 label: "Birthday",
 required: true
 },
 {
 name: "phone",
 component: "PhoneField",
 label: "Phone"
 },
 {
 name: "submit",
 component: "ButtonField",
 label: "Submit",
 type: "submit",
 icon: "Send"
 }
 ]
};

This code adds the following fields to your form:

The Text component displays some markdown
The PersonNameField is used to capture a person’s first and last names
The DateField allows a user to choose a date using a slick date picker
The PhoneField uses an input mask and country codes to guide the user when entering a phone number
The SubmitField contains a Send icon and allows the user to submit the form via a click or by pressing enter

Now, let’s take a look at the code that renders the component and handles the submit event:

ReactDOM.render(
 <Component
 definition={definition}
 onSubmit={({ component }) => {
 alert(JSON.stringify(component.getValues()));
 }}
 />,
 document.getElementById("root")
);

That’s it!? Yup! The mson-react layer automatically knows how to render the form component. It uses pub/sub and Pure Components to keep the rendering up-to-date.

When there is no validation error and the user clicks the submit button, an event with the name equal to the button’s name is emitted. In our case, this event is called submit. Therefore, we can define a handler using the onSubmit property. To keep things simple, we just alert the user of the entered values. Typically you want to do something like contact an API, redirect, etc…

Basic Form 2

Now, let’s go a bit deeper into the CRUD with a different example:

The first thing you may notice are the validators in the definition:

validators: [
 {
 where: {
 "fields.email.value": "nope@example.com"
 },
 error: {
 field: "email",
 error: "must not be {{fields.email.value}}"
 }
 }
]

Each field has a default set of validators, e.g. the EmailField ensures that email addresses are in a valid format. You can also extend these validators for a particular field or even for an entire form. For example, you can prevent the user from entering nope@example.com.

Next, let’s take a look at the code that loads the initial values when the component is mounted:

onMount={({ component }) => {
 // Load any initial data, e.g. from an API
 component.setValues({
 id: "abc123",
 firstName: "Bob",
 lastName: "Marley",
 email: "bob@example.com"
 });
}}

This code could just as easily be replaced by code that retrieves the values from some API asynchronously.

Finally, we use a more sophisticated event listener to handle the submit action. In a real-world app, you’d probably want to send a request to an API to save the data. You would receive a response from this API. If you receive an error, e.g. the email address is already in use, you can present this error to the user:

onSubmit={({ component }) => {
 // TODO: Contact some API with the data
 console.log("submitting", component.getValues());

 // Simulate response from API saying that email address is already in use and report this
 // error to the user
 if (component.get("fields.email.value") === "taken@example.com") {
 component.set({ "fields.email.err": "already in use" });
 } else {
 // Everything was successful so redirect, show confirmation, etc...
 }
}}

Live Demo

This post only touches on a small piece of what you can do using MSON. Because MSON is a full-featured language, you can declare all types of cool components. If you are interested in seeing more live examples, check out the live demo.

Wrap It Up!

If you are using React and Material-UI, you can speed up your app development by autogenerating your forms from JSON. This can be particularly useful if you need to bootstrap an app quickly and don’t want to have to worry about building a UI from scratch. If you liked this post, please give it a clap or two. Happy autogenerating!

Creating a New Programming Language That Will Allow Anyone to Make Software

Tue, 31 Jul 2018 00:00:00 +0000

I’ve spent the last 7 months working on a new programming language called MSON. Here is a nerdy post about why I did it, how it works and where I want to take it.

There are already a bazillion programming languages, why do we need another?

Software continues to eat the world and software development tools are evolving at lightning speeds, but it stills takes an in-depth knowledge of a programming language to develop software. Sure, there are millions of open-source projects that can be mashed together and robust cloud platforms that make the process easier, but it takes considerable skill to get all of these technologies working together. When I envision the future, I imagine the average person being able to make software, just like how the average person can create a spreadsheet today.

Creating a spreadsheet has become commonplace and this is largely a result of the intuitive user interface (UI) in MS Excel. Developing software; however, still remains something that is only done by a relative few and using complex programming languages. One could argue that it is somewhat trivial to create a UI that generates code, but it is almost infeasible to reverse the process and digest coded logic, as most programming languages have an almost infinite number of ways of representing logic. In other words, we don’t usually know how a piece of software will execute until we actually execute it. Creating a truly intuitive way of developing software requires a new paradigm, a new language that is easily read, modified and written by a UI.

It was back in 2003, when PHP was the hotness, that I started constructing a framework called FORMfields. At the time, FORMfields was pretty cool as you could create forms with PHP objects and then FORMfields would automatically generate the corresponding HTML, CSS and JS. Soon after creating FORMfields, I co-founded a company called GoExpo that produces event management software that is powered by FORMfields. GoExpo grew to process millions of dollars in booth sales annually for some of the largest trade shows in the world and this was a testament to the effectiveness of FORMfields.

Over the years, FORMfields continued to be essential in allowing our team to create a consistent UI that could be modified quickly (remember, these were the days before Twitter Bootstrap and more modern UI frameworks). However, as browsers were becoming more capable, more logic was moving into the front end; and FORMfields, which is written in PHP, only runs in the back end.

Fast forward to 2018, where JavaScript is the workhorse of the web and the difference between the front end and back end can feel like night and day. GoExpo has been acquired and I now have time to revisit the notion of FORMfields with a more modern stack. I want the next generation of FORMfields to run in both the browser and the back end, be open source and become the foundation of something that can be used by the masses.

This line of thinking has given birth to MSON, an extensible declarative language that can be easily manipulated by a UI. MSON is comprised of just a few building blocks, yet it is as powerful as its non-declarative counterparts.

The purpose of this initial post is to introduce the core principles of the MSON programming language and elaborate on its unique design. I plan on writing a follow-up post later that will explain MSON from a layman’s perspective. This post; however, is rather technical as it is about the developer-layer stuff in MSON and not about the designed-for-all UI layer I’ll be writing on top. I will be elaborating on the following topics:

Declarative Syntax
Components, Validators, Events & Listeners
Access Control
Inheritance, Template Parameters & Composition
Aggregate Components
Schemas and Self Documentation
User-Defined JavaScript Components
Using MSON in Any JavaScript Code
Reusing MSON Code Throughout the Full Stack
In/Out Properties

I will then finish with a statement about how I plan to monetize MSON and make it a self-sustaining project. I’d love to hear what you think.

Language Principles

Declarative Syntax

MSON is short for Model Script Object Notation, which is intentionally similar to JSON (JavaScript Object Notation). In fact, MSON is a subset of JSON, so if you know JSON then you know the syntax of MSON!

Declarative languages are much easier for software to read and write as they define what the software must do without stating exactly how to do it. And JSON is a great foundation on which to build. It contains just a few main constructs, is ubiquitous and supported by a vast ecosystem.

Components

The smallest building block in MSON is called a component. Components maintain state and can also control presentation and are very similar to the components now commonplace in most web frameworks. Components can inherit, contain or wrap other components. The rendering layer supports plugins for different environments and the default plugin supports React and Material-UI. Use of the rendering layer is optional, so components can be used on both the front end and back end.

A simple form component used to collect a name and email address could look like:

{
 name: 'MyForm',
 component: 'Form',
 fields: [
 {
 name: 'name',
 component: 'TextField',
 label: 'Name',
 required: true
 },
 {
 name: 'email',
 component: 'EmailField',
 label: 'Email'
 },
 {
 name: 'submit',
 component: 'ButtonField',
 label: 'Submit',
 icon: 'CheckCircle'
 }
 ]
}

And when it renders it looks like:

And, real-time validation is automatically baked in:

The majority of the remaining examples in this post will focus on form components, as they are simple to visualize, but MSON can support any type of component, e.g. menus, snackbars, redirects, etc… In addition, you can use JavaScript to create user-defined components that can pretty much do anything else you can imagine.

Validators

Each field has a default set of validators, e.g. the EmailField ensures that email addresses are in a valid format. You can also extend these validators for a particular field or even for an entire form.

For example, you can prevent the user from entering nope@example.com:

{
 name: 'MyForm',
 component: 'Form',
 fields: ...,
 validators: [
 {
 where: {
 'fields.email.value': 'nope@example.com'
 },
 error: {
 field: 'email',
 error: 'must not be {{fields.email.value}}'
 }
 }
 ]
}

Template parameters like {{fields.email.value}} can be used to inject the values of fields. And, you can use any MongoDB-style query in the where. For example, if you had password and retypePassword fields, you could ensure that they are equivalent with:

where: {
 'retypePassword.fields.value': {
 $ne: '{{fields.password.value}}'
 },
 error: ...
}

Events & Listeners

Changes to properties in a component generate events and you can create listeners that respond to these events with actions. There are basic actions that set, emit, email, contact APIs, etc… and custom actions can also be built using JavaScript.

The following example sets the value of the email field based on the value supplied in the name field when the user clicks the submit button:

{
 name: 'MyForm',
 component: 'Form',
 fields: ...,
 validators: ...,
 listeners: [
 {
 event: 'submit',
 actions: [
 {
 component: 'Set',
 name: 'fields.email.value',
 value: '{{fields.name.value}}@example.com'
 }
 ]
 }
 ]
}

We can also make this action conditional, e.g. only set the email if it is blank:

listeners: [
 {
 event: 'submit',
 actions: [
 {
 component: 'Set',
 if: {
 'fields.email': {
 $or: [
 {
 value: null
 },
 {
 value: ''
 }
 ]
 }
 },
 name: 'fields.email.value',
 value: '{{fields.name.value}}@example.com'
 }
 ]
 }
]

And sometimes we want to nest actions so that a condition is met before all actions are executed:

listeners: [
 {
 event: 'submit',
 actions: [
 {
 component: 'Action',
 if: {
 'fields.email': {
 $or: [
 {
 value: null
 },
 {
 value: ''
 }
 ]
 }
 },
 actions: [
 {
 component: 'Set',
 name: 'fields.email.value',
 value: '{{fields.name.value}}@example.com'
 },
 {
 component: 'Set',
 name: 'fields.name.value',
 value: '{{fields.name.value}} Builder'
 }
 ]
 }
 ]
 }
]

Access Control

Unlike most programming languages, access control is a first-class citizen in MSON, so its easy to use without a lot of work. Access can be restricted at the form or field layers for the create, read, update and archive operations. (MSON is designed to encourage data archiving instead of deletion so that data can be restored when it is accidentally archived. You can, of course, permanently delete data when needed).

Each user can have any number of user-defined roles and access is then limited to users with specified roles. There is also a system role of owner that is defined for the owner of the data. Field-layer access is checked first and if it is missing it will cascade to checking the form-layer access. When the access is undefined at the form layer (and not defined at the field-layer), all users have access.

Here is an example configuration:

{
 name: 'MyForm',
 component: 'Form',
 fields: ...,
 validators: ...,
 listeners: ...,
 access: {
 form: {
 create: ['admin', 'manager'],
 read: ['admin', 'employee'],
 update: ['admin', 'owner', 'manager'],
 archive: ['admin']
 },
 fields: {
 name: {
 create: ['admin'],
 update: ['owner']
 }
 }
 }
}

Among other things, only users with the admin or manager roles can create records. In addition, only owners of a record can modify the name.

Inheritance

Inheritance is used to add additional functionality to a component. For example, we can extend MyForm and add a phone number:

{
 name: 'MyFormExtended',
 component: 'MyForm',
 fields: [
 {
 name: 'phone',
 component: 'PhoneField',
 label: 'Phone Number',
 before: 'submit'
 }
 ]
}

We can define new validators, listeners, access, etc… at this new layer. For example, we can pre-populate some data, lay out all fields on the same line and disable the email field by creating a listener for the create event:

{
 name: 'MyFormExtended',
 component: 'MyForm',
 fields: ...,
 listeners: [
 {
 event: 'create',
 actions: [
 {
 component: 'Set',
 name: 'value',
 value: {
 name: 'Bob Builder',
 email: 'bob@example.com',
 phone: '(206)-123-4567'
 }
 },
 {
 component: 'Set',
 name: 'fields.name.block',
 value: false
 },
 {
 component: 'Set',
 name: 'fields.email.block',
 value: false
 },
 {
 component: 'Set',
 name: 'fields.email.disabled',
 value: true
 }
 ]
 }
 ]
}

which would render as:

Template Parameters

Template parameters are helpful when creating reusable components as they allow you to make pieces of your component dynamic. For example, let’s say that we want our first field and the label of our second field to be dynamic:

{
 name: 'MyTemplatedForm',
 component: 'Form',
 fields: [
 '{{firstField}}',
 {
 name: 'secondField',
 label: '{{secondFieldLabel}}',
 component: 'EmailField'
 }
 ]
}

we can then extend MyTemplatedForm and fill in the pieces:

{
 name: 'MyFilledTemplatedForm',
 component: 'MyTemplatedForm',
 firstField: {
 name: 'firstName',
 component: 'TextField',
 label: 'First Name'
 },
 secondFieldLabel: 'Email Address'
}

Composition

The componentToWrap property allows you to wrap components, enabling your reusable components to transform any component. For example, we can use composition to create a reusable component that adds a phone number:

{
 name: 'AddPhone',
 component: 'Form',
 componentToWrap: '{{baseForm}}',
 fields: [
 {
 name: 'phone',
 component: 'PhoneField',
 label: 'Phone Number',
 before: 'submit'
 }
 ]
}

and then pass in a component to be wrapped:

{
 name: 'MyFormWithPhone',
 component: 'AddPhone',
 baseForm: {
 component: 'MyForm'
 }
}

You can even extend wrapped components, paving the way for a rich ecosystem of aggregate components comprised of other components.

Aggregate Components

MSON ships with a number of aggregate components such as the RecordEditor and RecordList, which make it easy to turn your form components into editable UIs with just a few lines of code.

Let’s define a user component:

{
 name: 'MyAccount',
 component: 'Form',
 fields: [
 {
 name: 'firstName',
 component: 'TextField',
 label: 'First Name'
 }, 
 {
 name: 'lastName',
 component: 'TextField',
 label: 'Last Name'
 },
 {
 name: 'email',
 component: 'EmailField',
 label: 'Email'
 }
 ]
}

we can then use a RecordEditor to allow the user to edit her/his account:

{
 name: 'MyAccountEditor',
 component: 'RecordEditor',
 baseForm: {
 component: 'MyAccount'
 },
 label: 'Account'
}

rendering as:

and if you were to click the Edit button you’d see:

You can also use the RecordList to display an editable list of these accounts:

{
 name: 'MyAccountsList',
 component: 'RecordList',
 label: 'Accounts',
 baseFormFactory: {
 component: 'Factory',
 product: { 
 component: 'MyAccount'
 }
 }
}

rendering as:

Schemas and Self Documentation

Schemas must be defined for all components, which means that MSON is strongly typed. For example, a schema that defines boolean and date properties may look like:

{
 name: 'MyComponent',
 component: 'Component',
 schema: {
 component: 'Form',
 fields: [
 {
 name: 'hidden',
 component: 'BooleanField',
 help: 'Whether or not the component is hidden'
 },
 {
 name: 'updatedAt',
 component: 'DateTimeField',
 required: true, 
 help: 'When the component was updated'
 }
 ]
 }
}

Schemas can also contain documentation via help properties, which means that components are self-documenting! In addition, schemas are inherited and can be overwritten to allow for more or even less constraints.

User-Defined JavaScript Components

The MSON compiler is written in JavaScript and can run in both the browser and in Node.js. As such, you can use any custom JS, including external JS libraries, to create your own components.

For example, here is a component that uses Moment.js to set a currentDay property to the current day:

import compiler from 'mson/lib/compiler';
import Component from 'mson/lib/component';
import Form from 'mson/lib/form';
import { TextField } from 'mson/lib/fields';
import moment from 'moment';
class MyComponent extends Component {
 // className is needed as JS minification strips the constructor name
 className = 'MyComponent';

 create(props) {
 super.create(props);
 this.set({
 // Define a currentDay property
 schema: new Form(
 fields: [
 new TextField({
 name: 'currentDay'
 })
 ]
 ),
 // Default currentDay
 currentDay: moment().format('dddd')
 });
 }
}
compiler.registerComponent('MyComponent', MyComponent);

And then MyComponent can be used in any MSON code.

You can also do things like define custom asynchronous actions, e.g. one that POSTs form data:

import compiler from 'mson/lib/compiler';
import Action from 'mson/lib/actions/action';
import Form from 'mson/lib/form';
import { TextField } from 'mson/lib/fields';

class MyAction extends Action {
 className = 'MyAction';

 create(props) {
 super.create(props);
 this.set({
 schema: new Form(
 fields: [
 new TextField({
 name: 'foo'
 })
 ]
 )
 });
 }

 async act(props) {
 const form = new FormData();
 form.append('foo', this.get('foo'));
 const account = props.component;
 form.append('firstName', account.get('firstName');
 form.append('lastName', account.get('lastName');
 form.append('email', account.get('email');
 return fetch({
 'https://api.example.com',
 {
 method: 'POST',
 body: form
 }
 })
 }
}

compiler.registerComponent('MyAction', MyAction);

And then you can use this in your MSON code:

{
 name: 'MyAccountExtended',
 component: 'MyAccount',
 listeners: [
 {
 event: 'submit',
 actions: [
 {
 component: 'MyAction',
 foo: 'bar'
 }
 ]
 }
 ]
}

Using MSON in Any JavaScript Code

There is always parity between compiled and uncompiled components so that the same feature set is supported by both compiled and uncompiled code. For example, our same MyAccount component can also be defined as:

import Form from 'mson/lib/form';
import { TextField, Email } from 'mson/lib/fields';

class MyAccount extends Form {
 className = 'MyAccount';

 create(props) {
 super.create(props);
 this.set({
 fields: [
 new TextField({
 name: 'firstName',
 label: 'First Name'
 }),
 new TextField({
 name: 'lastName',
 label: 'Last Name'
 }),
 new EmailField({
 name: 'email',
 label: 'Email'
 })
 ]
 })
 }
}

In fact, converting MSON code to this type of code is basically what the compiler does. Although, the compiler doesn’t actually transpile MSON to JS, it merely instantiates JS code based on the MSON definitions.

Since all MSON code can be compiled to JS code, you can use MSON components in any JS code. For example, you can set some fields and validate the data:

import compiler from 'mson/lib/compiler';
// Compile the MyAccount component
const MyAccount = compiler.compile({
 component: 'MyAccount'
});
// Instantiate the JS class with a default value
const myAccount = new MyAccount({
 // Default values
 value: {
 firstName: 'Bob'
 }
});
// Set the remaining data
myAccount.set({
 lastName: 'Builder',
 email: 'invalid-email@'
});
// Make sure the values are valid
myAccount.validate();
if (myAccount.hasErr()) {
 console.log(myAccount.getErrs());
}

In other words, you can use MSON in your existing JS code to save time writing complex code. By declaring components in MSON, you’ll remove a lot of boilerplate code and reduce the possibility of bugs. You’ll also have code that has a standard structure and is framework agnostic. And this code doesn’t add any unneeded frameworks or back-end dependencies to your codebase.

Reusing MSON Code Throughout the Full Stack

MSON components can be shared by both the front end and back end, allowing for key logic to be written once and then reused. For example, the same form validation rules can be enforced in the browser and by your back-end API.

Moreover, actions can be limited to the backEnd or frontEnd, so that the same component can adjust according to the host environment. For example, you may want a contact form to send an email to the user when it is used on the back end, but only display a snackbar on the front end:

{
 component: 'Form',
 fields: [
 {
 name: 'email',
 component: 'EmailField',
 label: 'Email'
 },
 {
 name: 'message',
 component: 'TextField',
 label: 'Message'
 },
 {
 name: 'Submit',
 component: 'ButtonField',
 label: 'Submit'
 }
 ],
 listeners: [
 {
 event: 'submit',
 actions: [
 {
 // Send an email on the back end
 component: 'Email',
 layer: 'backEnd',
 from: '{{fields.email.value}}',
 to: 'noreply@example.com',
 subject: 'My message',
 body: '{{fields.message.value}}',
 // Detach so that user doesn't have to wait for email
 // to send
 detach: true
 },
 {
 // Display a message to the user on the front end
 component: 'Snackbar',
 layer: 'frontEnd',
 message: 'Thanks for the message'
 }
 ]
 }
 ]
}

In/Out Properties

Sometimes you want the presence of data, but don’t want it to be written or read from the back end. For example, your default user component may not allow for the password to be read or edited:

{
 name: 'MyUser',
 component: 'Form',
 fields: [
 {
 name: 'name',
 component: 'TextField',
 label: 'Name'
 },
 {
 name: 'email',
 component: 'EmailField',
 label: 'Email'
 },
 {
 name: 'password',
 component: 'PasswordField',
 label: 'Password',
 hidden: true,
 in: false,
 out: false
 }
 ]
}

However, your EditPasswordForm may need to allow such access:

{
 name: 'EditPasswordForm',
 component: 'MyUser',
 listeners: [
 {
 event: 'create',
 actions: [
 {
 // Hide all fields
 component: 'Set',
 name: 'hidden',
 value: true
 },
 {
 // Show password field
 component: 'Set',
 name: 'fields.password.hidden',
 value: false
 },
 {
 // Allow user to write password to the back end
 component: 'Set',
 name: 'fields.password.out',
 value: true
 }
 ]
 }
 ]
}

Next Steps

Before publishing this post, I had showed MSON to only a handful of people. This may seem odd for an open-source project, but it was intentional, as I wanted to flesh out the language before doing a wider release. I didn’t want people to start using something where the building blocks would be changing drastically. Therefore, I started with a basic prototype of the compiler and then started building a sample app written in MSON. This allowed me to use an iterative approach to see what works and what doesn’t work. And after several rewrites of the compiler, thank goodness for unit tests and a patient wife, I’ve been able to implement a full-featured app. I’m sure the language will continue to evolve, but I now feel confident that MSON is a solid foundation on which to build.

The MSON compiler is completely written in JS and runs in both the browser and on Node.js. The repo is hosted on github at https://github.com/redgeoff/mson — if you like the idea please give it a star.

I also plan on monetizing this idea by developing out mson.co, which will host a UI that anyone can use to make software. The UI will read, modify and write MSON behind the scenes and a back end will automatically be created based on your component definitions. And for developers, you’ll have the option of using as much of the mson.co stack as you wish, e.g. you could bootstrap your app using the MSON UI Builder and then replace/customize the front end with your favorite technologies (the mson.co back end is exposed via a GraphQL API). Or, you can use MSON on just the front end and create your own custom back-end API.

I hope you enjoyed reading this post and I would love to hear your thoughts.

Running a Free VPN Server on AWS

Mon, 24 Jul 2017 07:39:02 -0800

This tutorial will take you through the steps of setting up an EC2 instance that will run the OpenVPN Server. It will then cover how to grant and revoke access through the VPN Server.

AWS has an awesome firewall built into its core services which can easily be used to make sure that only certain ports are open to the outside world. One extra step that we can take is to run a VPN Server that serves as the gateway to our protected EC2 instances. We can then shutdown direct SSH access to our EC2 instances and also have the freedom to block access to our entire network just by revoking access via our VPN Server. The later is very useful if you need to revoke access for a former employee.

Step 1— Create the VPN Security Group

Overview: security groups allow your servers to communicate with each other in a private cloud while exposing specific ports to the world. We are going to create a security group to allow VPN access to our VPN Server. We will assume that all your other EC2 instances are members of the default security group and that the default security group does not allow access from the outside world.

From the EC2 dashboard, click Security Groups:

Click Create Security Group:

Enter a name and description of vpn and specify inbound rules on ports 22, 443, 943 and 1194. Note: the protocol for port 1194 is UDP.

Note: if the IP addresses that your team uses are static then you can add yet another layer of security by specifying that IP address range in the Source of your rules. However, you’ll want to leave the Source blank if you want your team to be able to connect from different IPs as they may be working from a hotel, home, cafe, etc…

Step 2 — Create the EC2 Instance

Return to the EC2 Dashboard and then click Launch Instance:

Select Ubuntu (you can of course select almost any other OS that runs OpenVPN, but this tutorial is tailored for Ubuntu)

Select t2.nano and click Review and Launch

On the next screen, click Edit security groups

Select the vpn and default security groups and click Review and Launch

Click Launch, choose your key pair and then click Launch Instances

Step 3 — Disable Source/Destination Check

From the list of instances, select the VPN instance and then Networking->Change Source/Dest. Check from the drop down menu. Then click Yes, Disable. This is needed as otherwise, your VPN server will not be able to connect to your other EC2 instances.

Step 4— Create an Elastic IP Address

Overview: when an EC2 instance is stopped and restarted, the Public IP address changes. We want the IP address of our VPN Server to remain static so we’ll use an Elastic IP Address.

From the E2c Dashboard, select Elastic IPs

Click Allocate new address

Click Allocate and then Close.

Make a note of your Elastic IP address as this will be the Public IP Address of your VPN Server.

Then select the Elastic IP and click Associate address from the drop down menu.

Select the EC2 instance you just created and click Associate.

Step 5— Install and Configure the OpenVPN Server

SSH into your VPN server:

$ ssh ubuntu@PUBLIC-IP-OF-VPN-SERVER

Download our helper scripts and set up a default config:

$ git clone https://github.com/redgeoff/openvpn-server-vagrant
$ cd openvpn-server-vagrant
$ cp config-default.sh config.sh

Edit config.sh and enter in your configuration. Note: PUBLIC_IP should be equal to the Elastic IP Address that you created above.

$ nano config.sh

Switch to root

$ sudo su -

Update Ubuntu and install OpenVPN. Note: you will be prompted twice and when you do, select Keep the local version currently installed

$ /home/ubuntu/openvpn-server-vagrant/ubuntu.sh
$ /home/ubuntu/openvpn-server-vagrant/openvpn.sh

At this point, the OpenVPN Server is running.

Step 6 — Add the Route

Routes must be added to the server so that your team’s clients know which traffic to route to the VPN Server.

You can determine the proper subnet by returning to your list of EC2 instances, clicking on a target instance and identifying the Private IP.

Your network will be the first 2 parts of the Private IP appended with zeros, e.g. 172.31.0.0

On the VPN Server edit /etc/openvpn/server.conf and add something like the following:

push "route 172.31.0.0 255.255.0.0"

Then restart the VPN Server with:

$ systemctl restart openvpn@server

Step 7— Grant Access to Your VPN

Note: We assume that you are still SSH’d into the VPN and logged in as root.

Run the following command and be sure to replace client below with a unique name for your user/client.

$ /home/ubuntu/openvpn-server-vagrant/add-client.sh client

You’ll then find a configuration file at

~/client-configs/files/client-name.ovpn

You will want to provide this file to the individual on your team who will be connecting to your VPN. SCP is handy for downloading this .ovpn file from your VPN Server.

Your team can use one of various VPN clients such as Tunnelblick (OS X) and OpenVPN (Linux, iOS, Android and Windows). After installing one of these clients they should be able to set up the VPN config just by double clicking on the .ovpn file.

Note: once connected to the VPN, your users will want to use the Private IPs of your EC2 instances. You’ll probably want to use Route 53 to create subdomain records that route to the Private IPs.

Step 8— Revoke Access to Your VPN

Note: We assume that you are still SSH’d into the VPN and logged in as root.

Run the following command and be sure to replace client below with a unique name for your user/client.

$ /home/ubuntu/openvpn-server-vagrant/revoke-full.sh client