We’re doing a water cooler talk today. Also, Allen can tell you how not to leak secrets, Michael knows how to work a spreadsheet, and Joe has been replaced by an AGI.

The full show notes for this episode are available at https://www.codingblocks.net/episode208.

Topics

• Want to score Vue.js London tickets? Tweet using both @CodingBlocks and #vuejs for a chance to win! (vue.js)
• How do you decide which projects are worth trying to convert into a money-making endeavor?
• Samsung ChatGPT sensitive information leaks (mashable.com)
• U.S Military Documents Leaked To Minecraft Discord Server (kotaku.com)
• Real-Time Analytics Podcast with Tim Berglund (podcasts.apple.com)
• CodeWhisperer from Amazon (aws.amazon.com)
• How much did GPT 3 cost? (pcguide.com)
• How much did GPT 4 cost? (medium.com)
• How much did Alpaca cost to train? (newatlas.com)
• Have any experience with Twilio? It’s work! (twilio.com)

Resources we like

• docker init is a tool (in beta) built into the latest Docker Desktop that you can use to get a leg up on your next project. It makes it easy to create docker files with best practices, as well as a docker-compose file to get you up and running. (docker.com)
• screen is an open-source powerful terminal multiplexer that allows users to create, manage, and switch between multiple terminal sessions, enabling seamless multitasking and persistent remote connections in a single window.
  • How to use gnu screen (linuxize.com)
  • tmux is a similar utility that some people prefer (github.com)
  • tmux vs screen (stackoverflow.com)
• The VIVO Universal Treadmill Desk Riser is an adjustable, ergonomic workspace solution designed to fit most treadmills, allowing users to seamlessly combine their work and exercise routines for a healthy, productive lifestyle. (amazon.com)
• The LifeSpan Fitness Under Desk Walking Treadmill is a compact, low-profile treadmill designed to fit under standing desks, enabling remote workers to maintain an active lifestyle by seamlessly integrating walking or light jogging into their daily work routine, promoting better health and increased productivity. (amazon.com)
• Kubernetes Network Policies are a set of rules that define how pods within a cluster can communicate with each other and with external resources, allowing administrators to enforce fine-grained access control and enhance the security of their containerized applications. (kubernetes.io)

We’ve got a new / old opening…Allen goes off / on script? Michael denies Joe the “swing” vote, and Joe is all in on AI assistance

Testing for concurrency issues is hard because it’s non-deterministic – basically you get unlucky due to the timing of things

Serializability

• The problems we’ve been discussing the past few episodes have been around since the 1970’s
• The answer is always – just use serializable isolation!
• Serializable isolation is the strongest isolation
  • The database prevents ALL race conditions
    • Even if transactions run in parallel, they’re guaranteed to act/result the same as if they had run one at a time, one after another
• If they’re so much better, why have/use weaker isolation levels?

Common Implementations

• Executing the transactions serially, actually
• Two phase locking – was one of the only real available solutions for several decades
• Optimistic concurrency control – things like serializable snapshot isolation
• We’ll be talking about these in terms of a single node database

Actual Serial Execution

• The easiest way to get rid of race conditions is to really just run things one after another – no concurrency
• This was only implemented for the first time around 2007 – prior the performance was too poor
  • This is truly a loop over transactions submitted to the db engine
• What changed to make it possible?
  • RAM became cheap enough to store entire active datasets in memory – when this is done transactions can execute much faster as you don’t have to wait to load the data from disk
  • DB designers concluded that most OLTP transactions are usually short-lived and make a small number of reads and writes – so they can be run on a consistent snapshot using snapshot isolation outside of the serial execution loop
• Used by VoltDB/HStore, Redis, and Datomic
  • Sometimes single-threaded systems can perform better than concurrent ones simply because there’s no locking
    • However, you’re bound by a single CPU core
    • Transactions will need to be set up differently than in typical concurrent systems

Encapsulating transactions in stored procedures

• They talked about how the early implementations in db’s had the intention of making the entire flow part of the transaction – to book a flight, a person would be shown a list of flights, they’d choose the one they want, and it’d be stored
  • The problem with that approach is it can take a long time for that flow to be completed
  • For that reason, web-applications limit transactions to a single web / http request
• There can still be situations where the transaction can occur with multiple interactions between the application and the database
  • From the application, query to see if the seat is still available on the flight…Ok, it is..now send another query to the db to update the seat to the customer…now query the db again to get any additional information
    • Doing it this way in a serial transaction db would be too slow because there’s too much network latency / waiting
• In a single-threaded serial transaction, everything must be done all at once in a stored procedure
  • Keeping everything in memory and providing the stored proc everything it needs ensures the transaction is fast without waiting for any network or disk IO
    • Great picture in the book in Figure 7-9 that describes this

Pros and Cons of Stored Procedures

• They’ve been part of the SQL standard since….1999
  • Sometimes get a bad rep
    • Each vendor’s implementation has their own language
    • The book mentions that the SQL language hasn’t kept up with other programming languages and look/are archaic in comparison
    • It’s hard to manage code stored on the database server
      • Harder to debug
      • More difficult to keep in source control
      • More difficult to test
      • More difficult to gather metrics for monitoring
    • Because db’s are typically shared by many applications or a LOT of application code, non-performant stored procedure code can cause massive problems – usually worse problems than poorly written application code
• These issues have and can be remedied
  • Modern serializable databases use regular programming languages
    • VoltDB – Java/Groovy
    • Datomic – Java/Clojure
    • Redis – Lua
• When the database is in memory and the transactions are single threaded, stored procedures can actually be quite good
  • Because there’s no IO / networking overhead, transactions can occur quickly on a single thread
• VoltDB also executes stored procedures for replication!
  • This means the stored procedures have to be deterministic – datetimes have to use deterministic apis

Partitioning

• As mentioned before, doing serial transactions means you are limited to a single core of a single CPU
  • Read only transactions could occur on a separate thread using snapshot isolation
  • If you need high write throughput, the single thread on a single core could be a problematic bottleneck
• This is where partitioning comes into play – if you can divvy your data up in a way that would allow transactions to stay within a single partition, then you’ll have the ability to linearly scale your CPU cores/threads to the number of partitions you have
  • If your transaction has to go across multiple partitions, then the stored procedure must ensure that each partition is handled appropriately to keep everything serialized properly
  • VoltDB can handle multiple partitions
    • Doing cross-partition writes is much slower than single partition writes – VoltDB reports 1k cross-partition writes per second
• Determining if transactions can occur on a single partition takes a bit of planning
  • key-value data is likely a single partition transaction
  • data with multiple secondary indexes will likely require cross-partition transactions

Resources We Like

• Designing Data Intensive Applicationshttps://www.codingblocks.net/get/designing-data-intensive-applications

Tips of the Episode

• Copilot Labs is an optional extension for Github Copilot that adds some nifty new features to VSCode with Copilot. It installs as a new sidebar icon and has 4 major features:
  • Code explanation – What does this block of code do? Does the code I wrote do what I think it does?
  • Code translation – Not familiar with a language you’re reading? Convert it to one that you do!
  • IDE brushes – Modify existing code using a variety of brushes like you would in an art program – Add Types, Fix bugs, improve readability, resilience, add documentation and it looks like there’s a way to add custom branches!
  • Test generation – JS and TS only right now
  https://githubnext.com/projects/copilot-labs/
• from AndrewEver wanted a Windows-like ALT+TAB experience on your Mac? Introducing AltTab for Mactab-macos.netlify.app/">https://alt-tab-macos.netlify.app/
• A reason to use the terminal in Visual Studio Code
  • Any operations like a git status that show a list of files are easy to ctrl / cmd click to open directly in the editor
• Bitwarden as a LastPass replacement – Less than 1/3 the pricehttps://bitwarden.com/pricing/https://www.lastpass.com/pricing

What are lost updates, and what can we do about them? Maybe we don’t do anything and accept the write skew? Also, Allen has sharp ears, Outlaw’s gort blah spotterfiles, and Joe is just thinking about breakfast.

The full show notes for this episode are available at https://www.codingblocks.net/episode206.

News

• Thank you for the amazing reviews!
  • iTunes: JomilyAnv
• Want to help us out? Leave us a review.

data-intensive-applications.jpg" alt="Designing Data Intensive Applications" class="wp-image-8330" width="251" height="329">Great book!

Preventing Lost Updates

• Last episode we talked about weak isolation, committed reads, and snapshot isolation
• There is one major problem we didn’t discuss called “The Lost Update Problem”
• Consider a read-modify-write transaction, now imagine two of them happening at the same time
• Even with snapshot isolation, it’s possible that read can happen for transaction A before B, but the write for A happens first
  • Incrementing/Decrementing values (counters, bank accounts)
  • Updating complex values (JSON for example)
  • CMS updates that send the full page as an update
• Solutions:
  • Atomic Writes – Some databases support atomic updates that effectively combine the read and write
    • Cursor Stability – locking the read object until the update is performed
    • Single Threading – Force all atomic operations to happen serially through a single thread
  • Explicit Locking
    • The application can be responsible for explicitly locking objects, placing responsibility in the devs hands
    • This makes sense in certain situations – imagine a multiplayer game where multiple players can move a shared object. It’s not enough to lock the data and then apply both updates in order since the shared game world can react. (ie: showing that the item is in use)

Detecting Lost Updates

• Locks can be tricky, what if we reused the snapshot mechanism we discussed before?
• We’re already keeping a record of the last transactionId to modify our data, and we know our current transactionId. What if we just failed any updates where our current transaction id was less than the transactionId of the last write to our data?
• This allows for naive application code, but also gives you fewer options…retry or give up
• Note: MySQL’s InnoDB’s Repeatable Read feature does not support this, so some argue it doesn’t qualify as snapshot isolation

What if you didn’t have transactions?

• If you didn’t have transactions, let alone a snapshot number, you could get similar behavior by doing a compare-and-set
• Example: update account set balance = 10 where balance = 9 and id = ABC
• This works best in simple databases that support atomic updates, but not great with snapshot isolation
• Note: it’s up to the application code to check that updates were successful – Updating 0 records is not an error

Conflict resolution and replication

• We haven’t talked much about replicas lately, how do we handle lost updates when we have multiple copies of data on multiple nodes?
• Compare-and-Set strategies and locking strategies assume a single up-to-date copy of the data….uh oh
• The options are limited here, so the strategy is to accept the writes and have an application process to decide what to do
  • Merge: Some operations, like incrementing a counter, can be safely merged. Riak has special datatypes for these
  • Last Write Wins: This is a common solution. It’s simple but inaccurate. Also the most common solution.

Write Skew and Phantoms

• Write skew – when a race condition occurs that allows writes to different records to take place at the same time that violates a state constraint
  • The example given in the book is the on-call doctor rotation
  • If one record had been modified after another record’s transaction had been completed, the race condition would not have taken place
  • write-skew is a generalization of the lost update problem
• Preventing write-skew
  • Atomic single-object locks won’t work because there’s more than one object being updated
  • Snapshot isolation also doesn’t work in many implementations – SQL Server, PostgreSQL, Oracle, and MySQL won’t prevent write skew
    • Requires true serializable isolation
  • Most databases don’t allow you to create constraints on multiple objects but you may be able to work around this using triggers or materialized views as your constraint
  • They mention if you can’t use serializable isolation, your next best option may be to lock the rows for an update in a transaction meaning nothing else can access them while the transaction is open
• Phantoms causing write skew
  • Pattern
    • The query for some business requirement – ie there’s more than one doctor on call
    • The application decides what to do with the results from the query
    • If the application decides to go forward with the change, then an INSERT, UPDATE, or DELETE operation will occur that would change the outcome of the previous step’s Application decision
      • They mention the steps could occur in different orders, for instance, you could do the write operation first and then check to make sure it didn’t violate the business constraint
    • In the case of checking for records that meet some condition, you could do a SELECT FOR UPDATE and lock those rows
    • In the case that you’re querying for a condition by checking on records to exist, if they don’t exist there’s nothing to lock, so the SELECT FOR UPDATE won’t work and you get a phantom write – a write in one transaction changes the search result of a query in another transaction
• Snapshot isolation avoids phantoms in read-only queries, but can’t stop them in read-write transactions

Materializing conflicts

• The problem we mentioned with phantom is there’d no record/object to lock because it doesn’t exist
• What if you were to have a set of records that could be used for locking to alleviate the phantom writes?
  • Create records for every possible combination of conflicting events and only use those to lock when doing a write
    • “materializing conflicts” because you’re taking the phantom writes and turning them into lock records that will prevent those conflicts
      • This can be difficult and prone to errors trying to create all the combinations of locks AND this is a nasty leakage of your storage into your application
        • Should be a last resort

Resources We Like

• The 12 Factor App and Google Cloud (cloud.google.com)

Tip of the Week

• Docker’s Buildkit is their backend builder that replaces the “legacy” builder by adding new non-backward compatible functionality. The way you enable buildkit is a little awkward, either passing flags or setting variables as well as enabling the features per Dockerfile, but it’s worth it! One of the cool features is the “mount” flag that you can pass as part of a RUN statement to bring in files that are not persisted past that layer. This is great for efficiency and security. The “cache” type is great for utilizing Docker’s cache to save time in future builds. The “bind” type is nice for mounting files you only need temporarily. like source code in for a compiled language. The “secret” is great for temporarily bringing in environment variables without persisting them. Type “ssh” is similar to “secret”, but for sharing ssh keys. Finally “tmpfs” is similar to swap memory, using an in-memory file system that’s nice for temporarily storing data in primary memory as a file that doesn’t need to be persisted. (github.com)
• Did you know Google has a Google Cloud Architecture diagramming tool? It’s free and easy to use so give it a shot! (cloud.google.com)
• ChatGTP has an app for slack. It’s designed to deliver instant conversation summaries, research tools, and writing assistance. Is this the end of scrolling through hundreds of messages to catch up on whatever is happening? /chatgpt summarize (salesforce.com)
• Have you heard about ephemeral containers? It’s a convenient way to spin up temporary containers that let you inspect files in a pod and do other debugging activities. Great for, well, debugging! (kubernetes.io)

There’s this thing called ChatGPT you may have heard of. Is it the end for all software developers? Have we reached the epitome of mankind? Also, should you write your own or find a FOSS solution? That and much more as Allen gets redemption, Joe has a beautiful monologue, and Outlaw debates a monitor that is a thumb size larger than his current setup.

If you’re in a podcast player and would prefer to read it on the web, follow this link:https://www.codingblocks.net/episode205

News

• Thank you for the amazing reviews!
  • iTunes: MalTheWarlock, Abdullah Nafees, BarnabusNutslap
• Orlando Code Camp coming up Saturday March 25th
  • https://orlandocodecamp.com/

ChatGPT

• Is this the beginning or the end of software development as we know it?
• Are you using it for work? Does your work have an AI policy?
• OpenAI has recently announced a whopping 90% price reduction on their ChatGPT and Whisper APi calls
  • $.002 per 1000 ChatGPT tokens
  • $.006 per minute to Whisper
• You also get $5 in free credit in your first 3 months, so give it a shot!
• https://openai.com/pricing

Roll Your Own vs FOSS

• This probably isn’t the first time and it won’t be the last we ask the question – should you write your own version of something if there’s a good Free Open Source Software alternative out there?

Typed vs Untyped Languages

• Another topic that we’ve touched on over the years – which is better and why?
• Any considerations when working with teams of developers?
• What are the pros and cons of each?

Cloud Pricing

• If you’re spending a good amount of money in the cloud, you should probably talk to a sales rep for your given cloud and try to negotiate rates. You may be surprised how much you can save. And…you never know until you ask!

Outlaw has the Itch to get a new Monitor

• Is it worth upgrading from a 34″ ultrawide to a 38″ ultrawide?
• What’s a good size for a 4k monitor?
  • Should you even get a 4k monitor?
• Should you go curved?
• Some references mentioned during the show
  • NVidia monitor search page: https://www.nvidia.com/en-us/geforce/products/g-sync-monitors/specs/
  • LG 38″ ultrawide: https://amzn.to/3SLeqUO38.jpg" alt="LG 38 Ultrawide Monitor">
  • Rtings recommended gaming monitors: https://www.rtings.com/monitor/reviews/best/by-usage/gaming
  • Games Radar best G-Sync monitors: https://www.gamesradar.com/best-g-sync-monitors/
  • Acer Predator 38″ ultrawide: https://amzn.to/3ZBDb80
  • Samsung Odyssey Neo G9 49″ Ultrawide: https://amzn.to/3ZGMTpx49.jpg" alt="Samsung 49 Ultrawide">
  • LG 49WQ95C-W 49″ Ultrawide: https://amzn.to/3mk0TY549.jpg" alt="LG 49 Ultrawide">

Resources from this episode

• How to jailbreak ChatGPT – List of Prompts: https://www.mlyearning.org/how-to-jailbreak-chatgpt/
• Magazine stops accepting submissions due to bots: https://nypost.com/2023/02/22/sci-fi-magazine-not-accepting-submissions-due-to-bots/
• Stack Overflow bans ChatGPT answers: https://www.theverge.com/2022/12/5/23493932/chatgpt-ai-generated-answers-temporarily-banned-stack-overflow-llms-dangers
• ChatGPT detection tool already out: https://www.ctvnews.ca/sci-tech/cheaters-beware-chatgpt-maker-releases-ai-detection-tool-1.6253847

Tips of the Week

• Did you know that the handy, dandy application jq is great for formatting json AND it’s also Turing complete? You can do full on programming inside jq to make changes – conditionals, variables, math, filtering, mapping…it’s Turing Complete!https://stedolan.github.io/jq/
• Want to freshen up your space, but you just don’t have the vision? Give interiorai.com a chance, upload a picture of your room and give it a description. It works better than it should.
• You can sort your command line output when doing something like an lssort -k2 -b
• On macOS you can drag a non-fullscreen window to a fullscreen desktop
• When using the ls -l command in a terminal, that first numeric column shows the number of hard links to a file – meaning the number of names an inode has for that file
• Argument parser for Python 3 – makes parsing command line arguments a breeze and creates beautiful –help documentation to boot!https://docs.python.org/3/library/argparse.html
• .NET has an equivalent parser we’ve mentioned in the pasthttps://www.nuget.org/packages/NuGet.CommandLine

Ever wonder how database backups work if new data is coming in while the backup is running? Hang with us while we talk about that, while Allen doesn’t stand a chance, Outlaw is in love, and Joe forgets his radio voice.

The full show notes for this episode are available at https://www.codingblocks.net/episode204.

News

• Thanks for the great reviews!
  • Audible: Allison Williams
• Orlando Code Camp 2023 is coming up on March 25th 2023 (orlandocodecamp.com)

data-intensive-applications.jpg" alt="Designing Data Intensive Applications" class="wp-image-8330">The big, beautiful, boar book: Designing Data-Intensive Applications

Weak Isolation levels

• If two transactions don’t touch the same data, they can be run in parallel.
• Race conditions occur when two different processes are trying to modify and access or modify the same data at the same time.
• Concurrency bugs are hard to find and test for – it usually comes down to unlucky timing.
• Concurrency bugs can also be very difficult to understand because multiple parts of an application can be interacting with the database simultaneously and in unexpected ways.
• Single-user interactions with a database are hard enough, and when you have multiple interactions happening simultaneously, it makes it all much more difficult.
• Databases try to make it look like interactions happen one at a time for that very reason – to simplify the work for a developer.
  • Serializable isolation is a database guarantee that makes transactions look as if they happened serially – one after another.
• Isolation is not that simple in reality
  • Serializable isolation comes at a performance cost
    • For this reason, most databases choose not to use it
  • Most databases use weaker isolation levels to protect against some concurrency issues but not all of them
    • These aren’t just theoretical bugs
      • Have resulted in large financial losses
      • Investigations by financial auditors
      • Customer data corruption
• It’s been a common theme that “use a relational db if you’re doing financial transactions” – however, being that most db’s use weak isolation, that doesn’t guarantee things would have been perfect
  • For this reason – you should understand the various weak isolation (non-serial) levels

Read Committed

• Two guarantees
  • When reading from the database, you will only see data that has been committed (no dirty reads)
  • When writing to the database, you will only overwrite data that has been committed (no dirty writes)
    • A second write is delayed until the first write’s transaction has been committed or aborted
    • This does not protect against the incrementing race condition – ie. two processes read a value at the same time, id = 1, then process one increments that and saves it, so the value is 2. Now, process two (which had 1 in memory from the read) does its increment, and stores the value as 2 as well – the value should have been three but because it had an old id, it didn’t increment properly
      • Avoiding this is discussed later in “Preventing Lost Updates”

Snapshot Isolation and Repeatable Read

• Addresses read skew, an example of a non-repeatable read
  • The example given was a customer has two bank accounts, gets her balance of account A, and then some moments after a transfer of $100 from account 2 to account 1 the customer gets the balance for account B…customer has an old value from account 1 and a new value from account two, so it looks like the customer is missing $100.
    • This is acceptable in the read committed isolation as both account values were committed at the time of the reads.
    • How could this happen? Here’s a quick example…
      • Multiple queries were issued to get the different account values – get balance for account 1, get balance for account 2…behind the scenes, someone did a transfer from one account to the other
  • This is a very temporary state
• There are situations where this can’t happen
  • Backups, analytics queries and integrity checks
• Snapshot isolation is a typical solution to the problem
  • Transactions read from a consistent snapshot – meaning that a transaction is opened to read multiple values from a snapshot state of the database when the read first started
    • Very popular feature – supported by PostgreSQL, Oracle, SQL Server, MySQL with InnoDB

How is snapshot isolation accomplished?

• Usually use write locks to stop dirty writes
  • Reads never block writes, and writes never block reads
• Because there may be multiple transactions taking place at once, there may need to be multiple copies of database objects in play at once – this is referred to as multi-version concurrency control
• The difference between read committed and snapshot isolation is read committed will use a different snapshot for each read whereas snapshot isolation will use the same snapshot for multiple reads within the transaction
  • They show an example of PostgreSQL’s implementation
    • Found this README in Postgres – https://github.com/postgres/postgres/blob/master/src/backend/storage/lmgr/README-SSI
    • The implementation basically uses some metadata fields on a row – created_by and deleted_by fields which contain transaction id’s
      • If you were to delete a row, that deleted_by field is updated, the row isn’t actually deleted at that point in time, but garbage collection will pick it up later and remove it physically from the table – at a time when it’s deemed that it will no longer be accessed
      • Updates are converted to creates and deletes (similar to what you’d see if you’re familiar with triggers in something like SQL Server)

Visibility for seeing a consistent snapshot

• Consistent snapshots work by following these rules:
  • At the start of a transaction, a list of all transactions in progress are identified and ignored for any reads
  • Any writes made by transactions that were aborted are ignored
  • Any writes made by a newer transaction id are ignored
  • All other writes are available to read
• Another way of thinking about it – an object is visible if
  • A transaction that had created an object had already been completed BEFORE the reader transaction starts
  • An object is either not marked for deletion OR if it is marked for deletion, the transaction had not been completed at the time the read transaction started
• Because the database is never truly updating/deleting values in place, a number of running transactions can continue to function from snapshots of those objects with very small overhead

Snapshot isolation and indexes

• Considering what we mentioned about the database storing multiple snapshots of state, how does this work with indexes?
  • One way would be to have the index point to all the transaction ids and have them filtered out, and when garbage collection happens, remove those entries from the index as well
• Turns out, there are a lot of implementation details and performance thoughts to take into consideration depending on the database implementation
  • They gave an example of how things are done in PostgreSQL – if multiple versions of the same object can fit on the same page file, nothing is done to the index
  • Another approach used by CouchDB and others are to use an append-only / copy-on-write method that does not overwrite the existing page in the b-tree but rather creates a copy of the modified page. Then, a copy of each parent is made all the way up to the root page to point to the new pages. Any pages not impacted by the write operation don’t need to be touched
    • If that sounded like it was creating a new tree for every write in the append-only-B-tree, you’d be correct. By taking this approach, every individual root tree is a consistent shot of the database at that point in time
      • The benefit of this approach is you don’t have to filter anything because every root node has only the transactions that belong in that snapshot
      • You do need a background process to garbage collect and compact

Repeatable read and naming confusion

• Unfortunately, snapshot isolation is known by many names
  • Oracle calls it serializable
  • PostgreSQL and MySQL call it repeatable read
• Why is it not consistent naming? Because it’s not a part of SQL, but rather a part of System R’s 1975 isolation levels, it hadn’t been defined yet!
  • They HAD defined repeatable read which is very similar to snapshot isolation
• Unfortunately in relational databases, “repeatable read” doesn’t define what guarantees they really provide
• It was called out that there is a formal definition of a repeatable read but most implementations don’t meet the definition
• “Nobody really knows what repeatable read means”

Resources We Like

• Orlando Code Camp 2023 (orlandocodecamp.com)
• The 12 Factor App and Google Cloud (cloud.google.com)
• Martin Kleppmann’s website (martin.kleppmann.com)
• Dave Foster Wallace – This is Water (youtube)

Tip of the Week

• “Infinite Jest” is an interesting book, but it’s not a good audio experience. Get the physical book this time. (amazon.com)
• Tamara Makes Games is a game dev on YouTube that has a lot of videos oriented around isometric, city-builder, and strategy games similar to Factorio. It’s a cool niche, and it’s a nice balance of code and visuals that are a delight to watch. (youtube)
• There’s a lot you can do with iTerm2’s status bar, it’s highly configurable making it easy to show system resource monitors, shell information, and other miscellaneous items. (status-bar.html">iterm2.com)
• Google has a tool named “container-diff” for analyzing and comparing container images. It can examine images along several different criteria, great for tracking down issues..like knowing why Docker isn’t caching a layer. (github.com)
• Xeol is a great utility for checking for end-of-life packages that you should get rid of. Thanks, gaprogman! (github)
• Using minikube? You can manage the space for the vm by using minikube ssh to shell into the machine and then prune your images with docker builder prune. Alternatively, you can use eval $(minikube docker-env) it to proxy docker to your local machine so you can just docker builder prune (and any other docker commands) without the shell. (minikube.sigs.k8s.io)

It’s time we learn about multi-object transactions as we continue our journey into Designing Data-Intensive Applications, while Allen didn’t specifically have that thought, Joe took a marketing class, and Michael promised he wouldn’t cry.

The full show notes for this episode are available at https://www.codingblocks.net/episode203.

News

• Thanks for the reviews!
  • iTunes: Dom Bell 30, Tontonton2
• Want some swag? We got swag! (/swag)
• Orlando Codecamp 2023 is coming up in March 25th 2023 (orlandocodecamp.com)

Single Object and Multi-Object Operations

data-intensive-applications.jpg" alt="Designing Data Intensive Applications" class="wp-image-8330">Best book evarr!

• Multi-object transactions need to know which reads and writes are part of the same transaction.
  • In an RDBMS, this is typically handled by a unique transaction identifier managed by a transaction manager.
  • All statements between the BEGIN TRANSACTION and COMMIT TRANSACTION are part of that transaction.
• Many non-relational databases don’t have a way of grouping those statements together.
• Single object transactions must also be atomic and isolated.
• Reading values while in the process of writing updated values would yield really weird results.
  • It’s for this reason that nearly all databases must support single object atomicity and isolation.
  • Atomicity is achievable with a log for crash recovery.
  • Isolation is achieved by locking the object to be written.
• Some databases use a more complex atomic setup, such as an incrementer, eliminating the need for a read, modify, write cycle.
• Another operation used is a compare and set.
• These types of operations are useful for ensuring good writes when multiple clients are attempting to write the same object concurrently.
• Transactions are more typically known for grouping multiple object writes into a single operational unit

Need for multi object transactions

• Many distributed databases / datastores don’t have transactions because they are difficult to implement across partitions.
  • This can also cause problems for high performance or availability needs.
  • But there is no technical reason distributed transactions are not possible.
• The author poses the question in the book: “Do we even need transactions?”
  • The short answer is, yes sometimes, such as:
    • Relational database systems where rows in tables link to rows in other tables,
    • In non-relational systems when data is denormalized for “object” reasons, those records need to be updated in a single shot, or
    • Indexes against tables in relational databases need to be updated at the same time as the underlying records in the tables.
• These can be handled without database transactions, but error handling on the application side becomes much more difficult.
  • Lack of isolation can cause concurrency problems.

Handling errors and aborts

• ACID transactions that fail are easily retry-able.
• Some systems with leaderless replication follow the “best effort” basis. The database will do what it can, and if something fails in the middle, it’ll leave anything that was written, meaning it won’t undo anything it already finished.
  • This puts all the burden on the application to recover from an error or failure.
• The book calls out developers saying that we only like to think about the happy path and not worry about what happens when something goes wrong.
• The author also mentioned there are a number of ORM’s that don’t do transactions proud and rather than building in some retry functionality, if something goes wrong, it’ll just bubble an error up the stack, specifically calling out Rails ActiveRecord and Django.
• Even ACID transactions aren’t necessarily perfect.
  • What if a transaction actually succeeded but the notification to the client got interrupted and now the application thinks it needs to try again, and MIGHT actually write a duplicate?
  • If an error is due to “overload”, basically a condition that will continue to error constantly, this could cause an unnecessary load of retries against the database.
  • Retrying may be pointless if there are network errors occurring.
  • Retrying something that will always yield an error is also pointless, such as a constraint violation.
  • There may be situations where your transactions trigger other actions, such as emails, SMS messages, etc. and in those situations you wouldn’t want to send new notifications every time you retry a transaction as it might generate a lot of noise.
    • When dealing with multiple systems such as the previous example, you may want to use something called a two-phase commit.

Tip of the Week

• Manything is an app that lets you use your old devices as security cameras. You install the app on your old phone or tablet, hit record, and configure motion detection. A much easier and cheaper option than ordering a camera! (apps.apple.com, play.google.com)
• The Linux Foundation offers training and certifications. Many great training courses, some free, some paid. There’s a nice Introduction to Kubernetes course you can try, and any money you do spend is going to a good place! (training.linuxfoundation.org)
• Kubernetes has recommendations for common-labels. The labels are helpful and standardization makes it easier to write tooling and queries around them. (kubernetes.io)
• Markdown Presentation for Visual Studio Code, thanks for the tip Nathan V! Marp lets you create slideshows from markdown in Visual Studio Code and helps you separate your content from the format. It looks great and it’s easy to version and re-use the data! (team.marp-vscode">marketplace.visualstudio.com)

We decided to knock the dust off our copies of Designing Data-Intensive Applications to learn about transactions while Michael is full of solutions, Allen isn’t deterred by Cheater McCheaterton, and Joe realizes wurds iz hard.

The full show notes for this episode are available at https://www.codingblocks.net/episode202.

News

• Thanks for the reviews!
  • iTunes: Jla115, Cuttin’ Corner Barbershop, mirgeee, JackUnver
  • Audible: Mr. William M. Davies
• Want some swag? We got swag! (/swag)

data-intensive-applications.jpg" alt="Designing Data Intensive Applications" class="wp-image-8330">It’s baaaaack!

Chapter 7: Transactions

• Great statement from one of the creators of Google’s Spanner where the general idea is that it’s better to have transactions as an available feature even if it has performance issues and let developers decide if the performance is worth the tradeoff, rather than not having transactions and putting all that complexity on the developer.
• Number of things that can go wrong during database interactions:
  • DB software or underlying hardware could fail during a write,
  • An application that uses the DB might crash in the middle of a series of operations,
  • Network problems could arise,
  • Multiple writes to the same records from multiple places causing race conditions,
  • Reads could happen to partially updated data which may not make sense, and/or
  • Race conditions between clients could cause weird problems.
• “Reliable” systems can handle those situations and ensure they don’t cause catastrophic failures, but making a system “reliable” is a lot of work.
• Transactions are what have been used for decades to address those issues.
  • A transaction is a way to group all related reads and writes into a single operation.
  • Either a transaction as a whole completes successfully as a “commit” or fails as an “abort, rollback”.
    • If the transaction fails, the application can choose what to do, like retry for example.
• In general, transactions make error handling much simpler for an application.
  • That was their purpose, to make developing against a database much simpler.
• Not all applications need transactions.
• In some cases, it makes sense not to use transactions for performance and/or availability reasons.

How do you know if you need a transaction?

• What are the safety guarantees?
• What are the costs of using them?

Concepts of a transaction

• Most relational DBs support transactions and some non-relational DBs support transactions.
• The general idea of a transaction has been around mostly unchanged for over 40 years, originally introduced in IBM System R, the first relational database.
• With the introduction of a lot of the NoSQL (non-relational) databases, transactions were left out.
  • In some NoSQL implementations, they redefined what a transaction meant with a weaker set of guarantees.
    • A popular belief was put out there that transactions meant anti-scalable.
    • Another popular belief was that to have a “serious” database, it had to have transactions.
      • The book calls out both as hyperbole.
      • The reality is there are tradeoffs for both having or not having transactions.
• ACID is the acronym to describe the safety guarantees of databases and stands for Atomicity, Consistency, Isolation, and Durability.
  • Coined in 1983 by Theo Harder and Andreas Reuter.
  • The reality is that each database’s implementation of ACID may be very different.
    • Lots of ambiguity for what Isolation means.
    • Because ACID doesn’t specify the actual guarantees, it’s basically a marketing term.
• Systems that don’t support ACID are often referred to as BASE, BAsically available, Soft state, and Eventual consistency.
  • Even more vague than ACID! BASE, more or less, just means anything but ACID.

Atomicity

• Atomicity refers to something that can not be broken into smaller parts.
  • In terms of multi-threaded programming, this means you can only see the state of something before or after a complete operation and nothing in-between.
  • In the world of database and ACID, atomicity has nothing to do with concurrency. For instance, if multiple actions are trying to processes the same data, that’s covered under Isolation.
    • Instead, ACID describes what should happen if there is a fault while performing multiple related writes.
      • For example, if a group of related writes are to be performed in an operation and there is some underlying error that occurs before the transaction of writes can be committed, then the operation is aborted and any writes that occurred during that operation must be undone, i.e. rolled back.
• Without atomicity, it is difficult to know what part of the operation completed and what failed.
• The benefit of the rollback is you don’t have to have any special logic in your application to figure out how to get back to the original state. You can just simply try again because the transaction took care of the cleanup for you.
  • This ability to get rid of any writes after an abort is basically what the atomicity is all about.

Consistency

• In ACID, consistency just means the database is in a good state.
• But consistency is a property of the application as it’s what defines the invariants for its operations.
  • This means that you must write your application transactions properly to satisfy the invariants that have been defined.
  • The database can take care of certain invariants, such as foreign key constraints and uniqueness constraints, but otherwise it’s left up to the application to set up the transactions properly.
  • The book suggests that because the consistency is on the application’s shoulders, the C shouldn’t be part of ACID.

Isolation

• Isolation is all about handling concurrency problems and race conditions.
  • The author provided an example of two clients trying to increment a single database counter concurrently, the value should have gone from 3 to 5, but only went to 4 because there was a race condition.
• Isolation means that the transactions are isolated from each other so the previous example cannot happen.
  • The book doesn’t dive deep on various forms of isolation implementations here as they go deeper in later sections, however one that was brought up was treating every transaction as if it was a serial transaction. The problem with this is there is a rather severe performance hit for forcing everything serially.
    • The section that describes the additional isolation levels is “Weak Isolation Levels”.

Durability

• Durability just means that once the database has committed a write, the data will not be forgotten, even if a database failure or hardware failure occurs.
  • This notion of durability typically means, in a single node database, that the data has been written to the drive, typically to a write-ahead log or similar implementation.
    • The write-ahead log ensures if there is any data corruption in the database, that it can be rebuilt, if necessary.
• In a replicated database, durability means that the data has been written to the other nodes successfully.
  • The performance implication here is that for the database to guarantee that it’s durable, it must wait for those distributed writes to complete before committing the transaction.
• PERFECT DURABILITY DOES NOT EXIST.
  • If all your databases and backups somehow got destroyed at the same time, there’s absolutely nothing you could do.

Resources we Like

• Coding Blocks Jam ’23 (itch.io)
• NewSQL (Wikipedia)
• Visual Studio (Wikipedia)
• Chrissy’s Court (IMDb)
• Tracy Morgan gets in a crash right after buying a $2 million Bugatti (CNN)
• IBM System R (Wikipedia)
• Database Schema for Multiple Types of Products (Coding Blocks)
• Uber’s Big Data Platform: 100+ Petabytes with Minute Latency (Uber)
• How to store data for 1,000 years (BBC)
• Longevity of Recordable CDs, DVDs and Blu-rays – Canadian Conservation Institute (CCI) Notes 19/1 (recordable-cds-dvds.html">canada.ca)

Tip of the Week

• The Bad Plus is an instrumental band that makes amazing music that’s perfect for programming. It’s a little wild, and a little strange. Maybe like Radiohead, but a saxophone instead of Thom Yorke? Maybe? (YouTube)
  • Correction, Piano Rock will quickly become your new favorite channel. (YouTube)
• docker builder is a command prefix that you can use that specifically operates against the builder. For example you can prune the builder’s cache without wiping out your local cache. It can really save your bacon if you’re working with a lot of images. (docs.docker.com)
• Ever want to convert YAML to JSON so you can see nesting issues easier? There’s a VSCode plugin for that! Search for hilleer.yaml-plus-json or find it on GitHub. (GitHub)
• Spotify has a great interface, but Apple Audio has lossless audio, sounds great, and pays artists more. Give it a shot! If you sign up for Apple One you can get Apple Music, Apple TV+, Apple Arcade, Apple News+ and a lot more for one unified price. (Apple)

Michael spends the holidays changing his passwords, Joe forgot to cancel his subscriptions, and Allen’s busy playing Call of Duty: Modern Healthcare as we discuss the our 2023 resolutions.

The full show notes for this episode are available at https://www.codingblocks.net/episode201.

News

• Thanks for the reviews CourageousPotato, Billlhead, [JD]Milo!
  • Want to help us out? Leave us a review.
• Game Jam is coming up, January 20-23! (itch.io)
• Thoughts on LastPass?
  • Check out the encrypted fields, as figured out by a developer. (GitHub)
  • LastPass users: Your info and password vault data are now in hackers’ hands (Ars Technica)

Our 2023 Resolutions

Michael’s

• Learn Kotlin,
• Go deeper on streaming technologies, such as Kafka, Flink, and/or Kafka Connect, and
• Learn more music theory and techniques.

data-intensive-applications.jpg" alt="Designing Data Intensive Applications" class="wp-image-8330" width="228" height="299">Drink!

JZ’s

• Of course Joe has categorized his resolutions into the following areas: finances, health, personal development, and career management,
• Go deeper on Spring and streaming technologies, and
• Do more game dev and LeetCode.

Q&A Round 1

• What skills are opposite and which are adjacent that can be picked up this year?
  • Angular unit testing,
  • Front end development,
  • Spring,
  • Big data concepts and technologies
• Any books, courses, or certifications?
  • Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann (Amazon)
  • Certified Kubernetes Application Developer (CKAD) (cncf.io)

Allen’s

• Spend more time focusing on health and fun,
• Updating the About Us page with recent info,
• Go deeper on streaming technologies and conepts,
• Go deeper on big data concepts such as data lakes, and best practices, etc.,
• Get back into making content again, such as YouTube, and/or maybe presenting.

Q&A Round 2

• What do you want to avoid in 2023?
  • Less Jenkins,
  • Avoid piecemeal Spring upgrades,

2023 Predictions

• Data, privacy … do we need it?,
• New languages, frameworks,
• Generated content (Dalle-2, ChatGPT, Copilot), and
• AI ethics
  • ChatGPT Wrote My AP English Essay—and I Passed (WSJ)

Resources

• Designing Data Intensive Applications (Amazon)
• Orlando Code Camp (OrlandoCodeCamp.com)
• Atlanta Dev Con (AtlDevCon.com)

Tip of the Week

• You can pipe directly to Visual Studio Code (in bash anyway), much easier than outputting to a file and opening it in Code … especially if you end up accidentally checking it in!
  • Example: curl https://www.codingblocks.net | code -
• Is your trackpad not responding on your new(-ish) MacBook? Run a piece of paper around the edge to clean out any gunk. Also maybe avoid dripping BBQ sauce on it.
• How does the iOS MFA / Verification Code settings work? We want MFA, but we we’re tired of the runaround!
• Jump around – nope, not Kris Kross, great tip from Thiyagarajan – keeps track of your most “frecent” directories to make navigation easier (GitHub)
  • There’s a version for PowerShell too – thank you Brad Knowles! (GitHub)

We step back and look at how things have changed since we first started the show while Outlaw is dancing on tables, Allen really knows his movie monsters, and Joe’s math is on point.

The full show notes for this episode are available at https://www.codingblocks.net/episode200.

News

• Thanks for the review nickname222Apple<3!
  • Want to help us out? Leave us a review.
• Want Free stickers? Send us a SASE, instructions over at (/swag)
• Game Jam is coming up, January 20-23 (itch.io)

Favorite Episodes

• We Still Don’t Understand Open Source Licensing (#5)
• Comparing Git Workflows (#90)
• Git from the Bottom Up series (#195)
• Designing Data-Intensive Applications series (series)
• The DevOps Handbook series (series)
• The Imposters Handbook series (series)
• Boxing and Unboxing in .NET (#2)
• Docker for Developers (#80)
• Elasticsearch (#83)
• Show Recursion Show (#154)
• Why is Python Popular? (#152)
• Hierarchical database patterns (series)

Favorite Events

• NDC 2020 (#126)
• Atlanta Dev Con (atldevcon.com)
• Orlando Code Camp (orlandocodecamp.com)
• South Florida Code Camp (#SoFloDevCon)
• Tampa Code Camp (facebook.com)
• Game Jams

How things have changed since we started?

• Social media
• The technologies we use
• Our careers
• Show format
• Media consumption habits
• Any viewpoints that have changed?
• Technology
• We’ve wrapped up 9 years…how have we changed the most…why?
• Bonus: Buying a window with 3 huge tvs (youtube.com)

Top 3 things you’ve gotten out of it …

• Alphabetize all the things in your class
• A better understanding of DB technologies and the impact of their underlying data structures
• It’s forced us to study various topics …
• Amazing friends, community
• The application tier can / should be your most powerful
• Don’t make your tech-du-jour a hammer

Tip of the Week

• If you want to enable Markdown support, open a document in Google Docs, head over to the top of the screen, go to “Tools” then “Preferences” and enable “Automatically detect Markdown.” After that, you’re good to go..except this only works for the current doc. (techcrunch.com)
• Markdown Viewer is also a plugin for Chrome that lets you support .md files in Google Drive (workspace.google.com)
• DataGrip’s useless “error at position” messages are frustrating, but the IDE actually does give you the info you need. Check your cursor!
• Minikube’s “profile” feature makes it easy to swap between clusters. No more tearing down and rebuilding if you need to switch to a new task! (minikube.sigs.k8s.io)
• SQLforDevs.com has a free ebook: Next-Level Database Techniques for Developers. (sqlfordevs.com)
  • Thanks for the tip Mikerg!

We talk about career management and interview tips, pushing data contracts “left”, and our favorite dev books while Outlaw is [redacted], Joe’s trying to figure out how to hire junior devs, and Allen’s trying to screw some nails in.

The full show notes for this episode are available at https://www.codingblocks.net/episode199.

News

• Thanks for the reviews Ryan Barger and Amazon Customer!
  • Want to help us out? Leave us a review.
• The sign-up form for The 3rd Coding Blocks Game Jam is live! #cbjam
  • Check out videos from past years:
    • CBJAM ’22 (youtube)
    • CBJAM ’21 (youtube)
• Interesting article about AI potentially replacing recruiters at Amazon (vox.com)

From ‘Round the Water-Cooler

Why don’t companies want junior developers?

• You see a lot of advice out there for developers to get that first job, but what advice does the industry have to trying to hire and support them? …not much

How long do you need to stay at a job?

• What do you do if you’re worried about being a “job hopper”?

Data Contracts..moving left?

• Where should normalization happen? (Analytics Engineering Podcast)

Most impactful books we’ve covered on the show

• Find more on our resources page
• Joe
  1. Designing Data-Intensive Applications (Amazon)
  2. The DevOps Handbook (Amazon)
  3. Clean Architecture (Amazon)
• Allen
  1. Designing Data-Intensive Applications (Amazon)
  2. The Imposter’s Handbook (bigmachine.io)
  3. Clean Architecture (Amazon)

How do you prepare to interview for a company?

• Interviewing…know what the company is creating so you’ll have an idea of what challenges they may have technically and so you can look up how you might solve some of those problems

How do you decide when to bring in new tech?

• Right tool for the job – don’t always be jumping ship to the newest, shiniest thing – it might be you just need to augment your stack with a new piece of technology rather than thinking something new will solve ALL your problems

Tip of the Week

• Did you know Obsidian has a command palette similar to Code? Same short-cut (Cmd/Ctrl-P) as VS Code and it makes for a great learning curve! Don’t know how to make something italic? Cmd-P. Insert a template? Cmd-P. Pretty much anything you want to do, but don’t know how to do. Cmd P! (help.obsidian.md)
• Ghostery plugin for Firefox cuts down on ads and protects your privacy. Thanks for the tip Aaron Jeskie! (addons.mozilla.org)
• Amazing prank to play on Windows user, hit F-11 to full screen this website next time your co-worker or family member leaves their computer unlocked. Thanks Scott Harden! (fakeupdate.net)