[{"content":"Intro While exploring Cloudflare Durable Objects, Actor Model is a term that I never heard before - something that Cloudflare claims simplifies the development of distributed systems. So this post is an attempt to understand what the Actor Model is, how it works, and why it is useful.\nWhat is the Actor Model? The Actor Model is a design pattern that reimagines concurrency by breaking away from traditional shared-memory approaches. Rather than wrestling with locks and synchronised access, it structures applications as networks of autonomous entities called actors, each one a self-contained unit with its own internal world.\nThese entities operate under three fundamental principles:\nThey process incoming requests sequentially, Guard their internal data from outside interference, and Collaborate by dispatching messages that prompt others into action. This architecture eliminates the chaos of competing processes by ensuring each component works in complete isolation while maintaining clear communication pathways.\nThe result is a system that sidesteps the typical pitfalls of concurrent programming - no race conditions, no deadlocks, just clean separation of concerns where each piece knows exactly what it\u0026rsquo;s responsible for and how to ask others for help when needed.\nConsider the following diagram that illustrates the Actor Model with four actors, each with its own state and mailbox for incoming messages, following the three principles mentioned above:\ngraph TD subgraph \"Actor System\" A[\"Actor AState: count: 5\"] B[\"Actor BState: balance: 100\"] C[\"Actor CState: queue: [1,2,3]\"] D[\"Actor DState: status: active\"] end subgraph \"Message Flow\" A --\u003e|\"increment(3)\"| B B --\u003e|\"process(data)\"| C C --\u003e|\"notify(complete)\"| D D --\u003e|\"reset()\"| A A --\u003e|\"query(status)\"| D end subgraph \"Actor Mailboxes\" MA[\"📬 Mailbox AMessages queued\"] MB[\"📬 Mailbox BProcessing sequentially\"] MC[\"📬 Mailbox COne at a time\"] MD[\"📬 Mailbox DFIFO order\"] end A -.-\u003e MA B -.-\u003e MB C -.-\u003e MC D -.-\u003e MD style A fill:#e1f5fe style B fill:#f3e5f5 style C fill:#e8f5e8 style D fill:#fff3e0 style MA color:#fff style MB color:#fff style MC color:#fff style MD color:#fff The essence of this diagram is that each actor receives messages in its own mailbox and processes them one at a time, ensuring that the internal state of each actor is not directly accessible by others. An example flow of messages could be:\nActor A sends an increment(3) message to Actor B. Actor B processes this message and sends a process(data) message to Actor C. Actor C then sends a notify(complete) message to Actor D. Actor D responds with a reset() message back to Actor A. This model allows for clear communication and separation of concerns, making it easier to reason about the system\u0026rsquo;s behaviour.\nHere\u0026rsquo;s a simple TypeScript implementation of an actor that processes messages sequentially, demonstrating the principles of the Actor Model:\nclass Actor { private mailbox: Message[] = []; private state: any; constructor(initialState: any): void { this.state = initialState; } // Process messages sequentially public receive(message: Message): void { this.mailbox.push(message); this.processNext(); } private processNext(): void { if (this.mailbox.length \u0026gt; 0) { const message = this.mailbox.shift(); this.handleMessage(message); setTimeout(() =\u0026gt; this.processNext(), 0); // Simulate async processing } } private handleMessage(message: Message): void { // Handle the message and update state console.log(`Processing message: ${message.content}`); // Update state based on message this.state = { ...this.state, ...message.data }; } } Event-Driven Architecture? Actors taking to each other through messages? Isn\u0026rsquo;t that just an event-driven architecture? Not quite. While both share similarities in communication mechanism, and facilitate scalability and fault tolerance, Actors are stateful and encapsulate logic whereas event handlers are typically stateless and reactive. In addition, Actors Model typically have direct message passing between actors, whilst event driven systems events are often broadcast to multiple subscribers.\nAspect Actor Model Event-Driven Architecture Communication Actors send asynchronous messages (HTTPS/RPC calls) to each other. Components emit or listen to events, often via a broker or event bus. State Management Each actor encapsulates its own state and behavior. and behavior. State is typically externalised, with services reacting to event payloads. Concurrency Model Inherent; each actor can process messages independently. Concurrency is often managed at the event processing layer or through reactive programming. Processing Flow Message -\u0026gt; Actor -\u0026gt; internal logic/state update -\u0026gt; response. Event emitted -\u0026gt; 0 or more subscribers react to it. Frameworks Akka, Microsoft Orleans, Cloudflare Durable Objects. Apache Kafka, RabbitMQ, AWS EventBridge, Node.js (EventEmitter) Pattern Focus Strongly aligns with OOP and distributed computing. Often used in microservices, real-time systems, or reactive UIs. ⬇️ Differences \u0026mdash;\u0026mdash;- \u0026mdash;\u0026mdash;- Encapsulation Actors are stateful and encapsulate logic. Event handlers are often stateless. Routing Direct message passing to specific actors. Event broadcast or pub/sub with multiple consumers. Granularity Fine-grained; every actor is a micro-unit. Typically coarser-grained services or functions. Back to Cloudflare Durable Objects Durable Objects are Cloudflare\u0026rsquo;s take on actor-based design - running right at the edge, which effectively means they are distributed across Cloudflare\u0026rsquo;s global network to provide low-latency access so that each actor (Durable Object) can handle requests close to the user.\nEach Durable Object:\nHas a globally unique identifier, Maintains its own persistent and isolated state, Guarantees that only one instance handles messages at a time. A typical use case for Durable Objects is to manage chat rooms, where each room is represented by a Durable Object that maintains list of participants and chat history, and processes messages (e.g., new messages, user joins/leaves) sequentially.\nWhere This All Fits For today\u0026rsquo;s developers and DevOps engineers, the actor model is more than an academic concept-it’s a practical blueprint for building safe, scalable systems.\nDevelopers can use it to simplify real-time logic and avoid complex locking mechanisms. DevOps teams can benefit from clearer patterns for deploying distributed workloads. Cloud practitioners can better model workflows that combine messaging, compute, and persistence in an efficient, fault-tolerant way. And with tools like Cloudflare Durable Objects, the actor model is no longer something you have to build yourself - it\u0026rsquo;s ready to use out of the box.\nMore to Explore proto.actor: A cross-platform actor model framework. Akka: A toolkit for building concurrent, distributed, and resilient message-driven applications on the JVM. Building stateful actor services on AWS Cloudflare Durable Objects ","permalink":"https://ylweng.com/blog/2025-06-01-actor-model/","summary":"\u003ch2 id=\"intro\"\u003eIntro\u003c/h2\u003e\n\u003cp\u003eWhile exploring Cloudflare Durable Objects, \u003cstrong\u003eActor Model\u003c/strong\u003e is a term that I never heard before - something that Cloudflare claims simplifies the development of distributed systems. So this post is an attempt to understand what the Actor Model is, how it works, and why it is useful.\u003c/p\u003e\n\u003ch2 id=\"what-is-the-actor-model\"\u003eWhat is the Actor Model?\u003c/h2\u003e\n\u003cp\u003eThe Actor Model is a design pattern that reimagines \u003cem\u003econcurrency\u003c/em\u003e by breaking away from traditional shared-memory approaches. Rather than wrestling with locks and synchronised access, it structures applications as networks of autonomous entities called \u003cstrong\u003eactors\u003c/strong\u003e, each one a self-contained unit with its own internal world.\u003c/p\u003e","title":"Actor Model"},{"content":"Moving my personal website to Cloudflare Workers For a while, I have been hosting my personal website on GitLab Pages, which is a solid choice for static sites whilst the code is hosted on GitLab, and I appreciate the smooth CI/CD integration. However, I\u0026rsquo;ve always had an eye on Cloudflare, especially hearing about their generous free tier and the great performance of their edge computing. I\u0026rsquo;ve even used Cloudflare Pages for dev versions of my site, which was cool. Then I was searching for a custom domain for the website, and ended up on Cloudflare because it was the cheapest option I could find! After I landed on the Cloudflare dashboard, I started exploring their offerings and was impressed by the range of services they provide, with a few clicks I somehow opened the Cloudflare Workers dashboard. Turns out, they\u0026rsquo;re recommending Workers for all new projects now, even over Pages, because it’s got more features and is where the future development is happening.\nNew Workers Worker? So, I decided to give it a shot. Setup a new Worker was super easy using the Cloudflare UI, I just had to import the GitLab repository, configure some settings, and voilà, I had a new Worker running. Oh, not quiet, I had to add a wrangler.toml file to the repository, which is the configuration file for Cloudflare Worker:\nname = \u0026#34;web\u0026#34; compatibility_date = \u0026#34;2025-03-25\u0026#34; [assets] directory = \u0026#34;./static\u0026#34; What Cloudflare essentially does is added a webhook to the GitLab repository, so every time I push a commit or merged a pull request into the main branch, it automatically triggers a deployment build to the Worker. This is pretty similar to how GitLab Pages works, but with the added benefit of being able to run server-side code at the edge.\nCustom domain You can easily set up a custom domain for the Worker by doing two things:\nAdd the domain name to the Worker settings in the Cloudflare dashboard. Create a CNAME record in your DNS settings that points the (www) custom domain to the Worker URL. The CNAME record looks like this:\nName: www Type: CNAME Value: web.my_account.workers.dev TTL: Auto Proxied: True Image: CNAME record\nNew subdomain as a Worker My personal site isn’t just one thing; I’ve got various static projects living as subpaths. Moving these over was surprisingly straightforward. It basically involved creating a new Worker for each static site I wanted to host as a subdomain. The trick was tweaking the wrangler.toml file (that\u0026rsquo;s Cloudflare\u0026rsquo;s config file for Workers). I added specific routes patterns in there to send traffic for each subdomain to its own Worker.\nFor example, if I wanted to host a static site at wasm-python.ylweng.com, I’d set up the wrangler.toml like this:\nname = \u0026#34;wasm-python\u0026#34; compatibility_date = \u0026#34;2025-03-25\u0026#34; routes = [{ pattern = \u0026#34;wasm-python.ylweng.com/*\u0026#34;, zone_name=\u0026#34;ylweng.com\u0026#34; }] [assets] directory = \u0026#34;./public\u0026#34; After getting wrangler.toml sorted, the last piece was to add a CNAME record in Cloudflare DNS for each subdomain. The Name for the CNAME record was just the subdomain itself (like wasm-python), and the Target was set to the URL of the Cloudflare Worker, in this case wasm-python-playground.my_account.workers.dev. In this way, traffic gets routed exactly where it needs to go.\nZero Trust One of the cool things about Cloudflare is their Zero Trust security model. Everything is laid out in a way that makes it easy to secure your applications. For my personal site, I didn\u0026rsquo;t need to set up any complex security rules, but I did appreciate the option to add some basic protections like DDoS mitigation and bot management. It’s all pretty much set up by default, which is great for someone who might not be a security expert but still wants to keep their site safe.\nFinal thoughts Moving to Cloudflare has been a great experience so far. The performance is solid, the setup was straightforward, and end users are benefit from their edge computing. This means faster response times for users around the world (no more complains from friends on the other side of the earth!). Plus, the free tier is generous enough for my personal projects, so I’m not worried about costs for a while.\nOne thing I really look forward to is exploring more of what Cloudflare Workers can do. With the ability to run server-side code at the edge, I can start building more dynamic features into my site without having to manage a separate backend server. Though I\u0026rsquo;m a little frustrated with the build process, which is a blackbox at the moment, and the only thing you can do is pass the build command to the Worker, which is not very flexible. But admittedly, it is still in the Beta stage, so I hope they will improve it in the future.\n","permalink":"https://ylweng.com/blog/2025-05-31-gitlab-pages-to-cloudflare-workers/","summary":"\u003ch2 id=\"moving-my-personal-website-to-cloudflare-workers\"\u003eMoving my personal website to Cloudflare Workers\u003c/h2\u003e\n\u003cp\u003eFor a while, I have been hosting my personal website on GitLab Pages, which is a solid choice for static sites whilst the code is hosted on GitLab, and I appreciate the smooth CI/CD integration. However, I\u0026rsquo;ve always had an eye on Cloudflare, especially hearing about their generous free tier and the great performance of their edge computing. I\u0026rsquo;ve even used Cloudflare Pages for dev versions of my site, which was cool. Then I was searching for a custom domain for the website, and ended up on Cloudflare because it was the cheapest option I could find! After I landed on the Cloudflare dashboard, I started exploring their offerings and was impressed by the range of services they provide, with a few clicks I somehow opened the Cloudflare Workers dashboard. Turns out, they\u0026rsquo;re recommending Workers for all new projects now, even over Pages, because it’s got more features and is where the future development is happening.\u003c/p\u003e","title":"Cloudflare rocks"},{"content":"Why In a recent Django project, I needed to implement end-to-end testing using Selenium. While the setup worked smoothly locally, I wanted to containerize the development environment, including the test suites. However, I encountered a challenge when running the tests in Docker: Selenium requires a browser to be installed in the system, and not all Docker images come with a browser pre-installed.\nProblem version: \u0026#39;3\u0026#39; services: # django app: build: . command: python manage.py runserver 0.0.0.0:8080 volumes: # ... ports: - \u0026#39;8080:8080\u0026#39; # database db: # ... Writing a docker compose file for these services was quite straightforward, everything runs smoothly until I run the test suites. Selenium requires the chosen browser to be pre-installed in your system in order to perform the tests, however, not every docker image comes with a browser.\nSolution So there are two options:\nModify the Dockerfile of app to include the target browser Use the Selenium official docker image Whilst the first option can work, it might not be the most reusable and maintainable solution.\nSo here is the addition to the docker-compose.yaml file:\nservices: # ... # other services # ... # selenium firefox driver selenium-firefox: image: selenium/standalone-firefox shm_size: 2gb ports: - \u0026#39;4444:4444\u0026#39; - \u0026#39;7900:7900\u0026#39; restart: \u0026#39;no\u0026#39; The project is using LiveServerTestCase from the django.test package that automatically sets up a live development server on a random port, and the tests can make real HTTP requests to this server. Therefore, an important step for the end-to-end testing to work is to modify the host variable for any class using the LiveServerTestCase.\nfrom django.test import LiveServerTestCase class SeleniumAppTest(LiveServerTestCase): \u0026#34;\u0026#34;\u0026#34; Test my app! \u0026#34;\u0026#34;\u0026#34; host = \u0026#39;app\u0026#39; Since we are running tests inside the app container, we would like the Selenium to connect to the live development server on http://app:{random_port}.\n","permalink":"https://ylweng.com/blog/2023-07-20-selenium-in-container/","summary":"\u003ch2 id=\"why\"\u003eWhy\u003c/h2\u003e\n\u003cp\u003eIn a recent Django project, I needed to implement end-to-end testing using Selenium. While the setup worked smoothly locally, I wanted to containerize the development environment, including the test suites. However, I encountered a challenge when running the tests in Docker: Selenium requires a browser to be installed in the system, and not all Docker images come with a browser pre-installed.\u003c/p\u003e\n\u003ch2 id=\"problem\"\u003eProblem\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#ff6ac1\"\u003eversion\u003c/span\u003e: \u003cspan style=\"color:#5af78e\"\u003e\u0026#39;3\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#ff6ac1\"\u003eservices\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#78787e\"\u003e# django\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#ff6ac1\"\u003eapp\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#ff6ac1\"\u003ebuild\u003c/span\u003e: .\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#ff6ac1\"\u003ecommand\u003c/span\u003e: python manage.py runserver 0.0.0.0:8080\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#ff6ac1\"\u003evolumes\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e      \u003cspan style=\"color:#78787e\"\u003e# ...\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#ff6ac1\"\u003eports\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e      - \u003cspan style=\"color:#5af78e\"\u003e\u0026#39;8080:8080\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#78787e\"\u003e# database\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#ff6ac1\"\u003edb\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#78787e\"\u003e# ...\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eWriting a docker compose file for these services was quite straightforward, everything runs smoothly until I run the test suites. Selenium requires the chosen browser to be pre-installed in your system in order to perform the tests, however, not every docker image comes with a browser.\u003c/p\u003e","title":"End-to-End Testing with Docker and Selenium"},{"content":"Intro Recently, I have came across a book called Resilient Web Design by Jeremy Keith. If you are working with web and frontend technologies I\u0026rsquo;ll definitely recommended this unique handbook to you. A rather interesting point about this book is it doesn\u0026rsquo;t contain much technical details, but takes you through the history of World Wide Web and learn from the past. I have also found this service manual from GOV.UK very helpful and I would suggest you also take a look.\nMy main takeaways from this book summarised into:\nResilience is not just about infrastructures, communications, and recoverability. Build core features with simplest technologies, enhance UX progressively. Accessibility is important than consistency. Let each technology do its own thing. Thoughts The process of building a resilient web application should involving user experience and assess whether it has meet the minimum requirements of the users. Web is complex - due to many historic reasons, browsers implemented features in different ways so developers would need to test applications on different browser engines to ensure cross-browser compatibility. The application development will become tricky if the app is heavily rely on CSS and JavaScript where most features would work on some browsers but not on the others. Out of the three main components (HTML, CSS, JavaScript) of the Web, HTML is the most resilient layer - it works on all browsers (though some markup might not work in older, deprecated browsers) and is the foundation of websites. Without CSS or JavaScript, users would still be able to see websites in a less attractive way, provided that the HTML document is filled with contents on the server side.\nFrontend frameworks Before frontend libraries/frameworks like React and Angular was introduced about a decade ago, many websites are multi-page applications (MPA) where the HTML is built on the server side and get sent to client\u0026rsquo;s browser (aka, server side rendering). A new HTML document will be sent on each subsequent request from the user unless it is a AJAX request. Then React and other frameworks adopt the concept of single-page application (SPA) and shift the focus towards rendering pages directly on the client side using JavaScript and an small index.html document. SPAs allows developers to build interactive applications (like Twitter, Gmail, and Google Maps) offering great user experience, but it generally resulted in longer initial loading (due to the large JavaScript bundle size and the time which browser need to execute them), poor search engine optimisation, and higher requirements on user\u0026rsquo;s bandwidth and browsers. But what if the user has poor network bandwidth, or they have disabled or unable to run JavaScript on browser? This left the user with an empty screen and there\u0026rsquo;s nothing to fallback on.\nDoes that means we should not use SPAs at all? Simple answer is it depends. If it is a content focus website, then you might want to consider SSR frameworks, or even just a static site generator if there is no requirements on databases. On the other hand, one should definitely consider tools like React and Angular for a highly-interactive application. Regardless the route to go down, user experience should be placed on top of developer experience.\nBack to SSR There has been some new frameworks coming out in recent years claiming better performance, productivity, and developer experiences, despite the fundamental rendering strategy remains the same. However, there are some frameworks to watch for:\nRemix Astro Qwik A common strategy of all these frameworks are HTML-first (and less JavaScript!), meaning we\u0026rsquo;re back to server side rendering again. In addition, Remix also adopted progressive enhancement approach, which emphasis on accessibility of core features for all users, then focus on enhanced features which may only be to users with good network bandwidth and addition browser features.\nTakeaways Every user on the web should be able to access the web application, even if their browser might not support some features that have implemented in the app.\nBuild things with simplest and most suitable tech. Rather than building a button using a div element with a bunch of ARIA labels, CSS and JavaScript, it is preferable to use the native HTML button element with some simple CSS.\nLet each technology do its own thing - HTML is for rendering contents, CSS is for enhanced presentation, and Javascript is for additional cool features!\n","permalink":"https://ylweng.com/blog/2023-01-06-resilient-web-design/","summary":"\u003ch1 id=\"intro\"\u003eIntro\u003c/h1\u003e\n\u003cp\u003eRecently, I have came across a book called \u003cem\u003e\u003ca href=\"https://resilientwebdesign.com/\"\u003eResilient Web Design\u003c/a\u003e\u003c/em\u003e by\n\u003cem\u003eJeremy Keith\u003c/em\u003e. If you are working with web and frontend technologies I\u0026rsquo;ll definitely\nrecommended this unique handbook to you. A rather interesting point about this book is it\ndoesn\u0026rsquo;t contain much technical details, but takes you through the history of World Wide Web and\nlearn from the past. I have also found this \u003ca href=\"https://www.gov.uk/service-manual/technology/using-progressive-enhancement\"\u003eservice manual\u003c/a\u003e from GOV.UK\nvery helpful and I would suggest you also take a look.\u003c/p\u003e","title":"What makes a web application resilient?"},{"content":"Intro I have found this great article How browsers work on web.dev recently explaining how browsers work in detail. I\u0026rsquo;m using this blog post to help me make notes and somewhere that I can quickly refresh the memory. If you are working on Frontend-related stuff, I strongly recommend this article as it will help you understand browsers and how they work in the background and interact with HTML, CSS, and JavaScript.\nRole There are many browsers out there today and you are probably heard most of them: Chrome, Firefox, Edge, and Safari etc. The main function of a browser is to present whatever resources you have requested from a server. Generally it is a HTML document (with stylesheets and JavaScript), but with browser extensions it can render files in different format like PDF.\nArchitecture Figure: Browser\u0026rsquo;s high level structure (source: How browsers work)\nLet\u0026rsquo;s go through each component from top to bottom (then left to right) in the image:\nUser Interface - What you see now in the browser except from the content of this page, e.g. the refresh button, address bar, and extension icons. Browser engine - Handle and communicate actions in between the UI and the rendering engine. Rendering engine - Responsible for rendering requested resources, including tasks like parsing HTML and CSS and display the content (browsers like Chrome run one rendering engine instance per tab per process). Networking - This component handles all networks call like HTTP requests. Implementations are platform-specific based on a platform-independent interface. JavaScript Interpreter - For parsing and executing JavaScript code. UI Backend - Used for drawing the UI. Data Persistence - a persistence layer for store local data such as cookies, session/local storage, IndexedDB, WebSQL and FileSystem. The rest of the article focusing mainly on the rendering engine.\nRendering engine The rendering engine chapter contains detail about how the engine display HTML, CSS and images. Different browsers use different engines:\nInternet Explorer - Trident Firefox - Gecko Safari - Webkit Chrome - Blink (a fork of WebKit) Main flow Figure: Rendering engine main flow (source)\nA short summary of the flow:\n(Fetch data) - The rendering engine gets data (requested documents) from the networking layer in 8kB chunks. (Parse) - Parsing the HTML document and convert it into document object model (DOM) nodes (sounds familiar?) aka content tree or DOM tree. Styling in stylesheets (including external ones) will also be parsed into a CSS Object Model (CSSOM) tree. (Render Tree) - The content tree is then combine with the CSSOM tree to construct a render tree, which is the visual representation of the document in ordered tree nodes (this ensure the content is displayed in the correct order). (Layout) - this is a process of assigning exact coordinates of the screen to each node of the render tree so that each node is placed at the correct location according to the specified HTML and CSS styles, as well as their standards. (Painting) - Finally, the UI backend component goes through the render tree and \u0026ldquo;paint\u0026rdquo; each node to the user interface. Note that different rendering engine may have slightly different terminology for each step in the flow.\nParsing Parsers translates the document into a parse tree (representing the structure) based on syntax rules of the document.\nThere are two processes within parsing:\nlexical analysis - the lexer/tokeniser breakdown the input into tokens syntax analysis - the parser apply language syntax rules on the tokens to construct a parse tree The following image is an example of breakdown the expression 2 + 3 - 1 into a parse tree:\nThe parsing process is iterative - tokens are feed into the parser one by one and then try to find a match for the current token according to the grammar - vocabulary and syntax rules. If no rules are matched an exception will be raised, else add the token to the tree as a new node.\nFor more details on parsing refer to the parsing section.\nHTML parser A HTML parser is responsible for parsing the HTML markup into a parse tree. Refer to W3C HTML specification for HTML\u0026rsquo;s vocabulary and syntax rules.\nThe conventional parsers (discussed above) do not apply to HTML but they do work for CSS and JavaScript, this is because the grammar for HTML is not context-free grammar, and that\u0026rsquo;s why HTML still can render the document even if you missed some opening or closing tags. Given the amount of HTML pages around the world and the flexibility they have, it is extremely difficult and inconvenience to come up a CFG for HTML to use in browsers. In addition, this behaviour also means HTML can not be parsed using regular top bottom or bottom top parsers. See this section and beyond to know more details of how browser parse HTML into a DOM tree.\nCSS parser CSS can be parsed using context free grammar parsers, since there are already well defined rule sets for CSS.\nOrder of processing scripts and stylesheets Scripts (with the \u0026lt;script\u0026gt; tag) will be executed as soon as it is reached, if it is hosted somewhere else then it will be fetched from the network. The parsing process halts until the script is executed. There are two ways to prevent the halting behaviour:\nSet defer on the script, in which the script will be executed after parsing. Mark the script async in HTML so it would get parsed and executed on a different thread. Webkit and Firefox does an optimisation called speculative parsing where parsing and script execution are separated (off-load) to different threads for overall performance. This optimisation is only performed on external resources and doesn\u0026rsquo;t modify the DOM tree.\nWe can manipulate CSS using JavaScript therefore styles must be ready during the document parsing stage. Different browser has different strategy:\nFirefox - block all scripts until all stylesheets are loaded WebKit - block scripts only if the required styles is not ready Render tree The render tree (what you see in the browser inspector) is constructed alongside the DOM tree, it is a visual representation of the document which allows the browser to draw the elements defined in the document, and calculate the drawing position for each element depending on the page size. This means any changes to any attribute of the HTML elements will trigger the browser to re-render the entire DOM tree, which is a very expensive and slow operation (CPU-intensive) if a lot of updates are required.\nFrameworks To avoid re-render everything in the DOM tree, many notable frontend libraries and frameworks are introduced and trying to solve this problem:\nReact - Implemented a concept called virtual DOM which keeps a snapshot of the DOM tree in memory and any state change will trigger the generation of a new virtual DOM. React will compare the current virtual DOM with the real DOM, and calculate the difference. Any changes made to the virtual DOM will then reflected on the specific part of the real DOM tree later, without re-render the whole tree. The sync process between the virtual and the real DOM is called reconciliation. Angular - Also uses virtual DOM initially then moved to an additional concept called Incremental DOM. This technique does not create a in-memory virtual DOM tree but working on the real DOM tree directly at the expense of some speed to gain the advantage of less memory usage (which benefits mobile devices). The main idea is to compile the template (.html file) into a set of functions/instructions (as you would found in the repo) which builds and/or updates the current component of the app. Vue - Similar to React it implemented virtual DOM, but in addition to JSX it have templates which are also compiled into the virtual DOM render functions. Svelte - a relatively new framework which itself is a compiler that compiles components into JavaScript code during the build time and perform manipulation directly on the real DOM. However, this approach requires reassignment of references in order for the compiler to pick up these changes. SolidJS - Heavily inspired by React\u0026rsquo;s design philosophy but does not use virtual DOM. The idea is similar to Svelte - compile and transform templates into real DOM, which makes it one of most performant libraries/frameworks. The relationship between the render tree and the DOM tree is not one-to-one, e.g. elements with display: none will not appear in the render tree.\nFrames (Firefox) or renderer (WebKit) knows how to lay out and paint itself and its children, this is because each renderer represents a rectangular area corresponding to a node\u0026rsquo;s CSS box. WebKit has a special class RenderObject for the renderer.\nStyle computation is a big topic, refer to the following sections:\nThe flow of constructing the tree Style Computation Developer have put many efforts in style data sharing, style computation, and applying rules. Firefox have two trees for managing style context and style computation, where one is for rules, and the other one divides style contexts into structs containing styles for a particular category, e.g. div. Children of these structs inherits styles and only contains styles that are not in the parent node - saving memory as well as easy to construct a path from bottom of the tree all the way to the top.\nLayout Layout/reflow is the calculation of position and size for renderer when it is added to the tree.\nHTML uses a flow based layout model which can compute these values in almost one iteration, though HTML tables can require more iterations.\nLayout can be proceed left-to-right and top-to-bottom. The coordinate system is relative to the root frame and starts from top left (you might find this familar if you have used D3.JS before).\nThe process starts from the root renderer (html tag) and iteratively perform calculation for all renderers.\nPainting This stage uses the UI infrastructure component and traverse the render tree and call the paint() method of renderer for display content on the screen.\npainting order ","permalink":"https://ylweng.com/blog/2023-01-01-brief-note-on-browsers/","summary":"\u003ch2 id=\"intro\"\u003eIntro\u003c/h2\u003e\n\u003cp\u003eI have found this great article \u003cem\u003e\u003ca href=\"https://web.dev/howbrowserswork/\"\u003eHow browsers work\u003c/a\u003e\u003c/em\u003e on \u003cstrong\u003eweb.dev\u003c/strong\u003e recently explaining how browsers work in detail. I\u0026rsquo;m using this blog post to help me make notes and somewhere that I can quickly refresh the memory. If you are working on Frontend-related stuff, I strongly recommend this article as it will help you understand browsers and how they work in the background and interact with HTML, CSS, and JavaScript.\u003c/p\u003e","title":"Brief notes on how browsers work"},{"content":"Repost: case studies for the Music AI project.\nDiscover how our team improved an AI-driven music web application written in Django. Addressing usability issues, we introduced progress reporting, managed workload efficiently, and simplified melody navigation. Our decoupled frontend and backend enhanced development and integration with Svelte and Django Rest Framework. Embracing Celery for asynchronous tasks improved responsiveness and scalability. Google Sign-in bolstered authentication. Adopting best practices, we ensured code quality with Git, automated testing, and code review. Later, we have added monitoring and logging system by leverage the ELK stack - also known as Elasticsearch, Logstash, Kibana.\n","permalink":"https://ylweng.com/blog/2022-11-22-music-ai-project/","summary":"\u003cp\u003eRepost: \u003ca href=\"https://casestudiesrcg.blogspot.com/2022/11/asynchronous-task-scheduling-for-music.html\"\u003ecase studies\u003c/a\u003e for the Music AI project.\u003c/p\u003e\n\u003cp\u003eDiscover how our team improved an AI-driven music web application written in Django. Addressing usability issues, we introduced progress reporting, managed workload efficiently, and simplified melody navigation. Our decoupled frontend and backend enhanced development and integration with Svelte and Django Rest Framework. Embracing Celery for asynchronous tasks improved responsiveness and scalability. Google Sign-in bolstered authentication. Adopting best practices, we ensured code quality with Git, automated testing, and code review. Later, we have added monitoring and logging system by leverage the ELK stack - also known as Elasticsearch, Logstash, Kibana.\u003c/p\u003e","title":"Asynchronous task scheduling for the Music AI project"},{"content":"Previous problem I\u0026rsquo;ve used the Angular framework a lot in the past for multiple projects so that\u0026rsquo;s why I have given an attempt and used Scully (a static site generator for Angular) to build a personal website. In short, I have found the following problems:\nLighthouse performance score isn\u0026rsquo;t great, mainly due to bundle size, unused files, and the speed of contentful paint. Relatively smaller community when compared to other frameworks, thus less off the shelf plugins are available. Slightly inconvenient to organise markdown files and images together. By default you would put markdown files under some folders under the root directory, e.g. content/blog/, and place blog images under src/assets/images/, then in the markdown frontmatter you reference the images by path (same for referenced images in the markdown content). Whereas in Gatsby you can place images next to the markdown file or any location and you can reference images using relative paths, Gatsby will automatically handle the path for you during the build time. Of course, you can write a custom plugin in Scully to do the same thing. Not possible to integrate MDX - a file format based on markdown but also supports JSX. This means we can import/export react components (support for other frameworks might also exist) in mdx files which opens a lot of possibilities. Why Astro A couple of reasons:\nIt is fast. Astro serves HTML files and strips away unused JavaScript thus increasing client loading speed. It was designed for content-focus websites such as blogs, documentation and portfolios. Support for MDX :) UI-agnostic. This means I can write codes using different frameworks and mix them on the same page. For example, I can use React to create interactive visualisation (React has a much bigger visualisation ecosystem than other frameworks), and use Svelte to build my website to take advantage of its simplicity. Good documentation. Setting up an Astro project and finding out how to do things in Astro was a very good experience thanks to the documentation. Before I actually started to set up a project, I was able to form a rough idea how to replicate/migrate features from my past projects by just reading the documentation. If you\u0026rsquo;re interested, head to the docs to learn more.\nAlternatives There are other frameworks which can be used as SSG, here are some popular ones:\n11ty GatsbyJS Next.js ","permalink":"https://ylweng.com/blog/2022-10-02-moving-to-astro/","summary":"\u003ch2 id=\"previous-problem\"\u003ePrevious problem\u003c/h2\u003e\n\u003cp\u003eI\u0026rsquo;ve used the Angular framework a lot in the past for multiple projects so that\u0026rsquo;s why I have given an attempt and used Scully (a static site generator for Angular) to build a personal website. In short, I have found the following problems:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://web.dev/performance-scoring/\"\u003eLighthouse\u003c/a\u003e performance score isn\u0026rsquo;t great, mainly due to bundle size, unused files, and the speed of contentful paint.\u003c/li\u003e\n\u003cli\u003eRelatively smaller community when compared to other frameworks, thus less off the shelf plugins are available.\u003c/li\u003e\n\u003cli\u003eSlightly inconvenient to organise markdown files and images together. By default you would put markdown files under some folders under the root directory, e.g. \u003ccode\u003econtent/blog/\u003c/code\u003e, and place blog images under \u003ccode\u003esrc/assets/images/\u003c/code\u003e, then in the markdown frontmatter you reference the images by path (same for referenced images in the markdown content). Whereas in Gatsby you can place images next to the markdown file or any location and you can reference images using relative paths, Gatsby will automatically handle the path for you during the build time. Of course, you can write a custom plugin in Scully to do the same thing.\u003c/li\u003e\n\u003cli\u003eNot possible to integrate MDX - a file format based on markdown but also supports JSX. This means we can import/export react components (support for other frameworks might also exist) in mdx files which opens a lot of possibilities.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"why-astro\"\u003eWhy Astro\u003c/h2\u003e\n\u003cp\u003eA couple of reasons:\u003c/p\u003e","title":"Moving to Astro"},{"content":"TL;DR Angular is great for building single-page applications (SPA) that provides seamless transition between pages which enabled better navigation experiences for users. The way that SPA navigates to a different page is by rewrite the content in the index.html file (the only html file) with the content of the new page, unlikely traditionally we store each page as individual HTML file and serve to users on requests. One immediate disadvantage of SPAs is they are not SEO (search engine optimisation) friendly.\nSince there is only one HTML file it makes difficult for search engines to crawl the metadata information of other parts of the website other than the home page, a common solution is to rewrite each url\u0026rsquo;s website metadata such as page title, author, description, and many more metadata defined in the Open Graph Protocol. Each url should at least have the title and the description metadata inside the head tag:\n\u0026lt;meta name=\u0026#34;title\u0026#34; content=\u0026#34;SEO for angular apps\u0026#34; /\u0026gt; \u0026lt;meta name=\u0026#34;description\u0026#34; content=\u0026#34;Write a description so that potential visitors have better understanding of this page.\u0026#34; /\u0026gt; Although Scully (a static site generator for Angular) generates index.html files per page/route which helps search engines to crawl the contents, we still need a way to change the meta and the title programmatically for each page.\nI have discovered that there aren\u0026rsquo;t many off-the-shelf SEO packages on NPM available for Angular and packages like ngx-seo was published a year ago. On the other hand, I have also found that Angular is providing the platform-browser API and unsurprisingly it is the corner stone of these SEO packages. The platform-browser API has two classes Meta and Title which allows us to manage HTML meta tags and the title of a current HTML document. Sounds like a solution for what we try to achieve right?\nSEO service and component To keep things organised and reusable in the future, let\u0026rsquo;s create a new service for our SEO related functions. I have used the command ng g service shared/services/seo --skip-tests but you can create it anywhere to suit your need. In addition, I created a dedicated SEO component so that it can be reused across different pages:\nng g module shared/components/seo ng g component shared/components/seo/ -m seo Meta tags Beside the title and the description, we also need to find out important meta tags to be included in each page. I have selected the following tags but you can find out more on Open Graph Protocol and MDN.\ntitle (and universal title template) description keywords author og:title og:description og:type og:image og:url og:site_name (title if blog) twitter:card twitter:site twitter:creator twitter:title twitter:description twitter:image twitter:image:alt SEO service The SEO service basically is a wrapper around the Title and Meta services provided in @angular/platform-browser, we keep everything that interact with these services here to hide implementation details. To keep things simple I have use one function for multiple meta tags which could share the same meta. For example, I can use the blog description for metas description, og:description, and twitter:description.`\nimport { Injectable } from \u0026#39;@angular/core\u0026#39;; import { Title, Meta } from \u0026#39;@angular/platform-browser\u0026#39;; import { environment } from \u0026#39;src/environments/environment\u0026#39;; @Injectable({ providedIn: \u0026#39;root\u0026#39;, }) export class SeoService { constructor(private titleService: Title, private metaService: Meta) {} setPageMetas(metas: Metas): void { this.updateTitle(metas.title); this.updateDescription(metas.description); this.updateAuthors(metas.authors); this.updateKeywords(metas.keywords); this.updateOgMetas(metas.type, metas.site_name, metas.url); this.updateImage(metas.image); this.updateTwitterMetas(metas.twitterCard, metas.twitterCreator); } updateTitle(title: string | undefined): void { this.titleService.setTitle(title ?? \u0026#39;yldweng\u0026#39;); for (let meta of [\u0026#39;og:title\u0026#39;, \u0026#39;twitter:title\u0026#39;]) { this.metaService.updateTag({ name: meta, content: title ? `${title}${environment.titleTemplate}` : \u0026#39;yldweng\u0026#39; }); } } updateDescription(description: string | undefined): void { const metaArr = [\u0026#39;description\u0026#39;, \u0026#39;og:description\u0026#39;, \u0026#39;og:image:alt\u0026#39;, \u0026#39;twitter:description\u0026#39;]; for (let meta of metaArr) { this.metaService.updateTag({ name: meta, content: description ?? environment.description }); } } ... skip ... updateImage(image: string | undefined): void { for (let meta of [\u0026#39;image\u0026#39;, \u0026#39;og:image\u0026#39;, \u0026#39;twitter:image\u0026#39;]) { this.metaService.updateTag({ name: meta, content: image ? `${environment.url}${image}` : `${environment.url}${environment.image}` }); } } updateTwitterMetas( card: string | undefined, creator: string | undefined ): void { this.metaService.updateTag({ name: \u0026#39;twitter:card\u0026#39;, content: card ?? environment.twitterCard }); this.metaService.updateTag({ name: \u0026#39;twitter:creator\u0026#39;, content: creator ?? environment.twitterID }) } } export interface Metas { title?: string, description?: string, authors?: any[] | string, keywords?: any[] | string, type?: string, site_name?: string, url?: string, image?: string, twitterCard?: string, twitterCreator?: string } With this service, other services and components can call the setPageMetas function to create/update page metas through dependency injection.\nSEO component Although we could reuse the SEO service across other parts of the application, say, I have a blog post component and import the service to do whatever is needed. However, it is recommended to create a dedicated component for each feature whenever possible and delegate the application logic to services. This separation between components (user faced building block) and services allows you to produce maintainable application which is easy to refactor in the future.\nimport { Component, Input, OnInit, OnChanges, SimpleChanges, } from \u0026#39;@angular/core\u0026#39;; import { environment } from \u0026#39;src/environments/environment\u0026#39;; import { SeoService } from \u0026#39;@shared/services/seo.service\u0026#39;; @Component({ selector: \u0026#39;app-seo\u0026#39;, templateUrl: \u0026#39;./seo.component.html\u0026#39;, styleUrls: [\u0026#39;./seo.component.less\u0026#39;], }) export class SeoComponent implements OnInit, OnChanges { @Input() title: string | undefined = \u0026#39;yldweng\u0026#39;; @Input() description: string | undefined = environment.description; @Input() keywords: string[] | string | undefined = environment.keywords; @Input() authors: string[] | string | undefined = environment.author; @Input() image: string | undefined = environment.image; @Input() twitterCard: string | undefined = environment.twitterCard; @Input() type: string | undefined = \u0026#39;articles\u0026#39;; constructor(private seoService: SeoService) {} ngOnInit(): void { this.seoService.setPageMetas({ title: this.title, description: this.description, keywords: this.keywords, authors: this.authors, image: this.image, twitterCard: this.twitterCard, type: this.type, }); } /** * Execute when there are changes to Inputs * @param changes */ ngOnChanges(changes: SimpleChanges): void {} } Now we can import this component into other components to update metas! I have used the blog post component as an example:\n\u0026lt;app-seo *ngIf=\u0026#34;currentPost$ | async as currentPost\u0026#34; [title]=\u0026#34;currentPost.title\u0026#34; [description]=\u0026#34;currentPost[\u0026#39;description\u0026#39;]\u0026#34; [authors]=\u0026#34;currentPost[\u0026#39;authors\u0026#39;]\u0026#34; [image]=\u0026#34;currentPost[\u0026#39;thumbnail\u0026#39;]\u0026#34; [keywords]=\u0026#34;currentPost[\u0026#39;tags\u0026#39;]\u0026#34; type=\u0026#34;Blog\u0026#34; \u0026gt; \u0026lt;/app-seo\u0026gt; \u0026lt;!-- Other parts of the blog-post component --\u0026gt; Open the dev tools in the browser and I can see metas has been generated for my current blog page!\nReference Google Docs: SEO starter guide ","permalink":"https://ylweng.com/blog/2022-05-30-seo-for-angular-apps/","summary":"\u003ch2 id=\"tldr\"\u003eTL;DR\u003c/h2\u003e\n\u003cp\u003eAngular is great for building single-page applications (SPA) that provides seamless transition between pages which enabled better navigation experiences for users. The way that SPA navigates to a different page is by rewrite the content in the \u003ccode\u003eindex.html\u003c/code\u003e file (the only html file) with the content of the new page, unlikely traditionally we store each page as individual HTML file and serve to users on requests. One immediate disadvantage of SPAs is they are not SEO (search engine optimisation) friendly.\u003c/p\u003e","title":"SEO for angular apps"},{"content":"Recently, I have been working on a web application built from the Flask framework. Alongside regular development work for this project I have tested the app with various test methods such as unit testing, functional testing, and end-to-end testing. This is the very first time I have written tests in Python. In this blog post I will include some notes around the set up, how to write tests, and some thoughts on testing in Python vs JavaScript.\nGeneral Testing There are two popular Python testing frameworks - Pytest and Unittest. Pytest is easier to work with than the standard testing library Unittest, therefore I have chosen it for its simplicity so it can be pick up easily by other team members as well.\nSet up To get started, install pytest using pip or whatever method that suits your development environment. For me it was simple - we have containerised the environment so all I need to do is add pytest to the requirements.txt file and rebuild the docker image.\nTo keep test files organised I have created a tests folder under the root directory, in addition, I have also created the file tests/conftest.py which includes all the custom fixtures that I have defined. You can learn about fixtures in depth from the pytest docs, but on the high level fixtures are constants that can be used in any of the tests in the current directory or levels below relative to the conftest.py file. A fixture can be a dataset, a database object, a class, or just a string. There are predefined fixtures you may want to explore here. Here is a simple example of usage:\n@pytest.fixture def my_website_name(): return \u0026#34;yldweng\u0026#34; The output from the my_website_name() function then can be used in any test files by insert it into the function parameter list:\ndef test_website_url(my_website_name): assert website_url == \u0026#39;https://\u0026#39; + my_website_name + \u0026#39;.gitlab.io\u0026#39; Fixtures have an option parameter scope which you can control when to destroy or clean up the fixture. For example, if you have provided scope=\u0026quot;session\u0026quot; then this fixture will only be created and destroyed once, which is very useful for things like databases and external services that are used multiple times in a single test session. You can find out more about scopes here.\nAdd test Pytest finds test files with file names in the format test_*.py or *_test.py by default. To add a test, we need to define a function and pass any required fixtures to the function and include one or more logical comparisons by using the assert keyword. A simple example is the test_website_url function shown in the previous section.\nIn order to test a Flask app and sending requests without running a live server, we will need some fixtures looks like the following:\n@pytest.fixture(scope=\u0026#39;session\u0026#39;) def test_app(): \u0026#34;\u0026#34;\u0026#34; https://flask.palletsprojects.com/en/2.0.x/testing/ \u0026#34;\u0026#34;\u0026#34; os.environ[\u0026#39;FLASK_TESTING\u0026#39;] = \u0026#39;True\u0026#39; app = create_app() app.config.update({ \u0026#34;TESTING\u0026#34;: True, \u0026#34;WTF_CSRF_METHODS\u0026#34;: [], \u0026#39;CSRF_CHECK_REFERER\u0026#39;: False, # add any other config as required }) # other setup can go here yield app # clean up / reset resources here @pytest.fixture(scope=\u0026#39;module\u0026#39;) def test_client(test_app): \u0026#34;\u0026#34;\u0026#34; Create a test client using the Flask application configured for testing The test client makes requests to the application without running a live server \u0026#34;\u0026#34;\u0026#34; with test_app.test_client() as testing_client: # Establish an application context with test_app.app_context(): yield testing_client # Use this in other test files This test_client fixture is very useful and we can it to test requests:\ndef test_record_create_edit_delete(test_app, test_client, mock_record): \u0026#34;\u0026#34;\u0026#34; Create, edit and delete \u0026#34;\u0026#34;\u0026#34; response_create = test_client.post( \u0026#39;/record/create\u0026#39;, follow_redirects=True, data=mock_record ) assert response_create.status_code == 200 # edit response_edit = test_client.post( \u0026#39;/record/mock_record_1/edit\u0026#39;, follow_redirects=True, data=mock_record ) assert response_edit.status_code == 200 # edit non exist items response_edit = test_client.post( \u0026#39;/record/some_random_record/edit\u0026#39;, follow_redirects=True, data=mock_record ) assert response_edit.status_code == 404 # delete response = test_client.get(\u0026#39;/record/mock_record_1/delete\u0026#39;, follow_redirects=True) assert response.status_code == 200 assert b\u0026#34;Deleted mock_record_1\u0026#34; in response.data # delete collection that is not exist response = test_client.get(\u0026#39;/schema/mock_record_1/delete\u0026#39;, follow_redirects=True) assert response.status_code == 200 Things are so far so good when testing requests, but an extra step is required for testing functions that are called from views/commands, otherwise you will get this error - working outside of *** context. It could be request, session or application. These errors occurred because the function we are testing requires to interact with a particular context - where Flask stores data about HTTP requests and other important information. If the error complains about request and session then we need to pass these contexts to the test function, this will be the same for the application context.\nYou have already seen one example shown above - the test_client fixture uses the application context app_context. For functions requires request and session context, use\nwith test_app.test_request_context(): End-to-End Testing For e2e tests I used the Selenium Python bindings to get access to the Selenium WebDriver and the standard library Unittest as the testing framework to run e2e tests (why not Pytest? I tried but there are a lot of glitches so I turned to the official examples which use Unittest). As a minimum requirement you need to install Selenium (preferably use pip) and any one of the drivers. Since our team are working on a containerised environment, I used the official Selenium docker image to spin up a remote Chrome WebDriver, and this allows us to connect the webdriver directly via a url.\nAfter setting up everything I have created a dedicated directory e2e under the test directory so that we can run all e2e tests just from that folder.\nA very basic example of an e2e test can be testing whether the page has the correct title. Let\u0026rsquo;s create a test_page_title.py file and add the following:\nimport unittest from selenium import webdriver class TestPageTitle(unittest.TestCase): __options = webdriver.ChromeOptions() @classmethod def setUpClass(cls): cls.__options.add_argument(\u0026#39;--headless\u0026#39;) cls.driver = webdriver.Remote( \u0026#34;http://selenium-chrome:4444\u0026#34;, options=cls.__options) def test_title(self): pageUrl = \u0026#34;https://google.co.uk\u0026#34; driver = self.driver driver.maximize_window() driver.get(pageUrl) assert \u0026#34;Google\u0026#34; in driver.title @classmethod def tearDownClass(cls): cls.driver.delete_all_cookies() cls.driver.quit() if __name__ == \u0026#34;__main__\u0026#34;: unittest.main() We can now use python -m unittest discover tests/e2e to run tests in the tests/e2e directory. unittest.TestCase is the basic block of unittest and you can add as many tests in the file as you like, though you could also use unittest.TestSuite() to aggregate multiple testCase into one collection.\nsetupClass and tearDownClass are class methods which are run once before and after all tests, respectively. There are also setUp and tearDown methods that are similar to the class methods but they are run once for every test in the current class. In this class we only defined a single test test_title that tests the title for Google UK.\nIf you have multiple test cases that require the same set up or tear down methods, then you may wish to consider to inheriting common set up file from different test cases.\nFor example we can create a setup.py file with the same class methods as shown above:\nimport unittest from selenium import webdriver class ChromeSetup(unittest.TestCase): \u0026#34;\u0026#34;\u0026#34; A class for setup options for Chrome and do cleanup afterward. Each classmethod is run once per test file. \u0026#34;\u0026#34;\u0026#34; __options = webdriver.ChromeOptions() # A class method called before tests in an individual class are run @classmethod def setUpClass(cls): # https://docs.python.org/3/library/functions.html#classmethod # If a class method is called for a derived class, the derived class object is passed as the implied first argument. cls.__options.add_argument(\u0026#39;--headless\u0026#39;) cls.driver = webdriver.Remote(\u0026#34;http://selenium-chrome:4444\u0026#34;, options=cls.__options) # A class method called after tests in an individual class have run. @classmethod def tearDownClass(cls): cls.driver.delete_all_cookies() cls.driver.quit() and re-use them in the test file:\nimport unittest from setup import ChromeSetup class TestPageTitle(ChromeSetup, unittest.TestCase): @classmethod def setUpClass(cls): super(TestPageTitle, cls).setUpClass() def test_title(self): pageUrl = \u0026#34;https://google.co.uk\u0026#34; driver = self.driver driver.maximize_window() driver.get(pageUrl) assert \u0026#34;Google\u0026#34; in driver.title @classmethod def tearDownClass(cls): super(TestPageTitle, cls).tearDownClass() if __name__ == \u0026#34;__main__\u0026#34;: unittest.main() Here we pass cls as the first argument to methods in the setup.py file.\nPython vs JavaScript? For me the transition of writing tests in JavaScript to Python was quite smooth because the fundamentals were more or less the same. However, one thing that I like about testing Python is you only need to use one assert function for any comparison, whereas in JavaScript you often need to lookup and find the right assert.someFunction or expect.someFunction. The difficulty of setting up and getting tests running are the same in Python and JavaScript and mostly depends on your proficiency with the language and frameworks/libraries you are using. The better you understand how the application is running from top to bottom, the quicker you can write test suites.\nI would like to summarise the process into few steps:\nDo some research and knowing what test runners, frameworks and tools are available for the programming language, in particular, are there already packages that integrates the test framework and frameworks you are using for the application (e.g. pytest-flask for Flask framework and pytest). Test frameworks often comes with their own test runner to help you run the tests, but that is not always the case with other tools like selenium and Chai (assertion library for JS), then you will need to select a test runner (like unittest and pytest in Python) to get tests running. Choose the test toolkit and follow the official get started tutorial, write some very basic tests and get familiar with how the framework does assertions. Make sure you also take a look at what configurations provided by the framework and what integrations are available. Learn about how to organise test cases for your chosen test framework and ways to define global constants, or fixtures. Additional things to look for: How to run or exclude specific test Hooks for running functions before or after tests How to handle asynchronous codes Dive deep into documentation and find out how to mock different components of your application (especially if you are building a web app). Start with tests at the lower level (unit tests), and work upwards to test components, and the system as a whole. Integrate the testing into CI/CD. Happy testing and don\u0026rsquo;t forget to document important information for your team! ","permalink":"https://ylweng.com/blog/2022-04-28-testing-flask-app/","summary":"\u003cp\u003eRecently, I have been working on a web application built from the \u003ca href=\"https://flask.palletsprojects.com/en/latest/\"\u003eFlask\u003c/a\u003e framework. Alongside regular development work for this project I have tested the app with various test methods such as unit testing, functional testing, and end-to-end testing. This is the very first time I have written tests in Python. In this blog post I will include some notes around the set up, how to write tests, and some thoughts on testing in Python vs JavaScript.\u003c/p\u003e","title":"Testing Flask App"},{"content":"In my twelve-factors methodology blog post I have mentioned Semantic Versioning (SemVer) which defines a set of rules on how the version should be incremented on new changes. A few weeks later I have been searching for tools which can generate changelog from commits and bump up the version in the package.json file for me. Unsurprisingly, there are many packages available for this and semantic-release stands out of the crowd and offers many features beyond what I am after. A rough workflow is as follows:\nCommit new changes to the dev branch. Merge new features into the main branch. CI/CD pipeline runs all tests on new codes. Semantic-release package analysis commit messages, increment the version number in the package.json file accordingly, update the changelog file, and make a new commit to include all files that have changed. Semantic-release package creates a new release on GitHub/GitLab. A new production is built and deployed. At the moment you can have automated releases pushed to NPM, GitHub, and GitLab, or all of them if you wish to. In this post, I will give a simple guide on how to set up this package in GitHub actions and GitLab CI/CD pipeline for any JavaScript project.\nPrerequisite You need to have the package manager npm installed. A GitHub/GitLab account. You are committed to write commit messages that follows certain convention, e.g. Conventional commits, Angular. GitHub The following steps will help you install and configure the semantic-release package for use in GitHub actions.\nInstall the package using npm install --save-dev semantic-release.\nInstall extra plugins:\nnpm install @semantic-release/git @semantic-release/changelog conventional-changelog-conventionalcommits -D Add a new file called release.config.js to the root directory containing the following (see this [link] for detailed configuration):\nmodule.exports = { defaultBranch: \u0026#39;main\u0026#39;, // change this to match your repo branches: [ \u0026#39;+([0-9])?(.{+([0-9]),x}).x\u0026#39;, \u0026#39;main\u0026#39;, \u0026#39;next\u0026#39;, \u0026#39;next-major\u0026#39;, { name: \u0026#39;beta\u0026#39;, prerelease: true }, { name: \u0026#39;alpha\u0026#39;, prerelease: true }, ], plugins: [ [ \u0026#39;@semantic-release/commit-analyzer\u0026#39;, { preset: \u0026#39;conventionalcommits\u0026#39;, // see the official manual for more details releaseRules: \u0026#39;./releaseRules.js\u0026#39;, // custom release rules parserOpts: { noteKeywords: [\u0026#39;BREAKING CHANGE\u0026#39;, \u0026#39;BREAKING CHANGES\u0026#39;], }, }, ], [ \u0026#39;@semantic-release/release-notes-generator\u0026#39;, { preset: \u0026#39;conventionalcommits\u0026#39;, presetConfig: { types: [ // map commit type to sections in the changelog. // If your commit message does not include any of these, it will not but included in the changelog { type: \u0026#39;feat\u0026#39;, section: \u0026#39;✨ Features\u0026#39;, hidden: false, // whether to show this type of changes in the changelog }, { type: \u0026#39;fix\u0026#39;, section: \u0026#39;🐛 Bug Fixes\u0026#39;, hidden: false, }, { type: \u0026#39;docs\u0026#39;, section: \u0026#39;📝 Documentation\u0026#39;, hidden: false, }, { type: \u0026#39;style\u0026#39;, section: \u0026#39;🎨 Styles\u0026#39;, hidden: false, }, { type: \u0026#39;refactor\u0026#39;, section: \u0026#39;♻️ Code Refactoring\u0026#39;, hidden: false, }, { type: \u0026#39;perf\u0026#39;, section: \u0026#39;⚡️ Performance Improvement\u0026#39;, hidden: false, }, { type: \u0026#39;test\u0026#39;, section: \u0026#39;✅ Testing\u0026#39;, hidden: false, }, { type: \u0026#39;build\u0026#39;, section: \u0026#39;🔨 Build/Dependencies\u0026#39;, hidden: false, }, { type: \u0026#39;ci\u0026#39;, section: \u0026#39;🔧 Continuous Integration\u0026#39;, hidden: false, }, { type: \u0026#39;chore\u0026#39;, hidden: true, }, ], }, }, ], [ \u0026#39;@semantic-release/changelog\u0026#39;, { changelogFile: \u0026#39;CHANGELOG.md\u0026#39;, // location of the changelog }, ], [ \u0026#39;@semantic-release/npm\u0026#39;, { npmPublish: false, // disabled if you are not publish package to NPM }, ], \u0026#39;@semantic-release/github\u0026#39;, [ \u0026#39;@semantic-release/git\u0026#39;, { assets: [ // files to include with the release commit \u0026#39;CHANGELOG.mdx\u0026#39;, \u0026#39;package.json\u0026#39;, \u0026#39;package-lock.json\u0026#39;, \u0026#39;npm-shrinkwrap.json\u0026#39;, ], }, ], ], }; Add another file releaseRules.js containing all release rules:\nexport default [ { type: \u0026#39;feat\u0026#39;, release: \u0026#39;minor\u0026#39; }, { type: \u0026#39;fix\u0026#39;, release: \u0026#39;patch\u0026#39; }, { type: \u0026#39;perf\u0026#39;, release: \u0026#39;patch\u0026#39; }, { type: \u0026#39;chore\u0026#39;, release: false }, { breaking: true, release: \u0026#39;major\u0026#39; }, { scope: \u0026#39;no-release\u0026#39;, release: false }, ]; Note that the version number is expressed as major.minor.patch.\nRun npx semantic-release --dry-run (or -d) to do a dry-run which skips the prepare, publish, success and fail steps, and to get a preview of the pending release. In this step you will get an error something like SemanticReleaseError: No GitHub token specified. The reason is obvious, we have included the plugin @semantic-release/github but didn\u0026rsquo;t pass a GitHub personal token. The semantic-release package is supposed to be used and run in the CI pipeline after all tests are passed. To pass a GitHub token simply run GH_TOKEN=\u0026lt;your_github_token\u0026gt; npx semantic-release -d.\nTo actually making releases from a local machine (not recommended) you can run GH_TOKEN=\u0026lt;your_github_token\u0026gt; npx semantic-release --no-ci (or --ci false) to skip Continuous Integration environment verifications.\nAdd npx semantic-release to your GitHub Actions deployment workflows:\n- name: Semantic Release run: npx semantic-release Some tips Make sure the repo url in the package.json file matches the one you are working on to avoid pushing to a different repository. The latest tag must be in the format of vx.x.x, otherwise the version number will start from v1.0.0. If you are using the husky package, add HUSKY=0 before npx semantic-release to disable it in CI. To set the author of releases, pass the built-in token {{ secrets.GITHUB_TOKEN }} to either GH_TOKEN or GITHUB_TOKEN variable in GitHub actions workflow files. To set the author \u0026amp; committer details, include the following environment variables: env: GIT_AUTHOR_NAME: \u0026#39;github-actions-bot\u0026#39; GIT_AUTHOR_EMAIL: \u0026#39;support+actions@github.com\u0026#39; GIT_COMMITTER_NAME: \u0026#39;github-actions-bot\u0026#39; GIT_COMMITTER_EMAIL: \u0026#39;support+actions@github.com\u0026#39; If you want to push to protected branches, make sure you have included the following in the workflow: - uses: actions/checkout@v3 with: ref: master persist-credentials: false ... - name: Semantic Release run: HUSKY=0 npx semantic-release env: GH_TOKEN: ${{ secrets.GH_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} // both GH token are included where GITHUB_TOKEN is the default one ... Once you have included this package in the CI/CD pipeline, there is no more you need to do other than creating more fancy features to your applications!\nGitLab The set up for GitLab CI/CD is identical to GitHub except the following:\nInstall @semantic-release/gitlab and replace '@semantic-release/github' with '@semantic-release/gitlab' in the release.config.js file. Add npx semantic-release to the .gitlab-ci.yml file. Result Click here to see an example of the end result, and the corresponding markdown file. In addition, if you have linked any issue or pull request in the commit message, the semantic-release package will also add a comment in the relevant issue/PR to say this is included in the new release. How exciting!\nOther languages? This package will also work with languages other than JavaScript, and there is an official guidance on this.\n","permalink":"https://ylweng.com/blog/2022-03-23-automated-version-management/","summary":"\u003cp\u003eIn my twelve-factors methodology blog post I have mentioned Semantic Versioning (\u003ca href=\"https://semver.org/\"\u003eSemVer\u003c/a\u003e) which defines a set of rules on how the version should be incremented on new changes. A few weeks later I have been searching for tools which can generate changelog from commits and bump up the version in the \u003ccode\u003epackage.json\u003c/code\u003e file for me. Unsurprisingly, there are many packages available for this and \u003ca href=\"https://github.com/semantic-release/semantic-release\"\u003esemantic-release\u003c/a\u003e stands out of the crowd and offers many features beyond what I am after. A rough workflow is as follows:\u003c/p\u003e","title":"Automated version management and changelog"},{"content":"This is my note for the event - \u0026lsquo;Building and testing apps for the Cloud\u0026rsquo; hosted by the Research Software Engineering (RSE) team at The Hartree Centre. I\u0026rsquo;m going to share what I have learned and some thought from this event, and you might find it useful and interesting if you are interested in the following:\nbest practice for building/testing cloud-native apps containerisation continuous integration, and continuous deployment Twelve-factor methodology The RSE team shared an industry-leading twelve-factor app methodology in which I have found really useful and beneficial. This methodology is specifically for designing and building software-as-a-service (SaaS) apps, but there are some factors which I think are also applicable in non-cloud environments. When I got introduced to this methodology, my immediate thought/question was: what are ideal ways to continuously deliver features without loss of efficiency, as the complexity and scale of the system increases? There is no golden approach to building a web application step by step, but this methodology is a very good place to start.\nNow let\u0026rsquo;s take a look at each of the twelve factors to get an idea of these principles.\nImage: a sketch of a 12-factors app\nCodebase - This principle states that you should have only one codebase even if you have multiple environments. This helps to maintain version control as everyone is working on the same repository, as well as keeping consistency across different environments. Dependencies - It is important to specify all dependencies for each individual app somewhere in the repository, however, dependencies should not include OS specific binaries that could lead to compatibility issues. Some well known dependency management tools are NPM for Javascript, RubyGem for Ruby, pip for Python, Composer for PHP, etc. I have also learned that JFROG is used widely in large software engineering teams or projects for artifact management, in this case it might be used for storing production files and dependencies packages. Config - Configurations (that vary between different environments) should be separate from code and injected into code logic when needed, i.e. they should not be declared as constants in the codebase. Instead, configurations and important credentials in twelve-factor apps are stored as secret/environment variables. This is very common on continuous integration tools. Backing Service - Any dependent components, or other application services such as databases, cache services, message queues, and third-party services, should be treated as external resources attached to the app, and accessed through network (URLs, or web addresses). For example, we can connect an Amazon RDS DB instance via mydb.123456789012.us-east-1.rds.amazonaws.com or an local MySQL database via mysql://...@host/db. This principle will allow any attached resources to be replaced without modifying the code logic of the app, creating a fault tolerance app with high availability. Now the question is, what if the dependent services goes down? Build, Release, Run - Before deploying the app, there are three stages that need to go through. During the build stage the codebase is used for compile executable binaries so the build artifact can be used elsewhere. The release stage combines the build artifact and the production environment configurations to produce a read-to-go release. And finally the run stage spins up everything and deploy the app (and dependencies) into the live environment. These stages are strictly separated and there are many benefits of automating this process using CI/CD. Processes - This factor stated that we should execute the app as one or more stateless processes and store any data (if required longer than the current session duration) in a stateful backing service such as databases or cache services. Therefore, sharing data between different processes of the app violates this factor, instead, use shared cache session instead of in-memory storage to prevent loss of data. Like me, you may find this factor difficult to understand initially, but keep in mind that this factor is essential for factor 8 and factor 9, as well as scalability of the application. Port Binding - This factor extends from the fourth factor (Backing service) and treats the app as a self-contained service which can be accessed via a URL by binding to a port. For non-cloud environments, port binding is often done by the webserver container such as Tomcat (port 8080) for Java and Node.js (port 3000) for Javascript. On the other hand, containers are widely used in cloud development and we can change and bind the port inside the dockerfile. You will see an example in the practical section soon. Concurrency - The app can scale out horizontally via the process model where we run multiple instances of the app at the same time, provided that the app does not violate the sixth factor. This means the app can have multiple workers that can handle more traffic and workload. Keep in mind that apps/backing services are far more fault tolerant on a horizontal scale than vertical scale. Disposability - This factor requires the app instance being able to be created or destroyed at any time with fast startup and graceful shutdown. In connection to the sixth factor, when a shutdown request is sent to the process (an running app instance), it should wait until all the current jobs are finished before cleaning up and exit; when the traffic increases, app instances should spin up very quickly to cope with the high workload. This page includes some process management tools which you may find interesting. Dev-Prod Parity - Keep development, staging, and production as similar as possible is the key of this factor, so the app should be designed for continuous deployment by keeping the gap between development and production to a minimum. The gap is not limited to time and tools, but also who is responsible for development and deployment. Logs - Treat logs as event streams. The app should continuously output time-ordered logs to stdout and/or stderr whenever it is running. Such logs will provide useful insight for various stakeholders. A good destination of logs will be Splunk. Admin processes - Any admin script should be separate from the app and its codebase, and stored in a separate repository. However, the admin processes should be run in the same environment of the app. Therefore, admin processes also need to go through the build, release, and run, three stages. It is worth mentioning that, in my experience, factors 1, 2, 3, 5, 10 and 11 are also important and applicable for non-cloud developments.\nThis section is only scratching the surface of the methodology here and I recommend you to read through the documentation in detail by visiting 12factor.net. If you want to learn more about twelve factors in depth, checkout Beyond the Twelve-Factor App which added three additional factors on top of the original 12 factors.\nPractical session I This practical session is all about the (local) deployment of a full stack application using Docker containers. The full tutorial is kindly made available by Jim and you can find it on Gitlab so I won\u0026rsquo;t repeat the whole process again. In short, this is a simple React (A Javascript library) application with a graph displaying the temperatures of cities and the app is updated every 5 seconds. In the backend there is a Postgres timescale database and a GraphQL server sitting on top of the database for GraphQL API access.\nSummarise as steps:\nInstall Docker and apply version control for all codebases (one for the frontend and another for the GraphQL server).\nStore credentials using echo \u0026quot;credential_value\u0026quot; | docker secret create credential_name -\nCreate a GraphQL server docker image from the GraphQL server repository using docker build -f Dockerfile.prod -t fullstack/graphql . where -f specifies the path of Dockerfile to build (or /Dockerfile by default) and -t specifies the name and tag of the image in the format name:tag. We can also create a test image for the server specifying the test procedure - docker build -f Dockerfile.test -t fullstack/graphql-test ..\nCreate a frontend docker production image from the React app repository using docker build -t fullstack/frontend ..\nCreate a load balance image from nginx (dockerfile).\nCreate a docker compose file docker-compose-fullstack.yaml:\n# Compose file specification version version: \u0026#39;3.5\u0026#39; services: # Create a timescale Postgres database from the image on Docker registry timescale: image: timescale/timescaledb:latest-pg12 # Expose this service on port 5432 (first number) # (map from port 5432 from inside the container) ports: - \u0026#39;5432:5432\u0026#39; # store the password securely environment: POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password # Specify what secrets to receive from the secret section secrets: - postgres_password # Location for permanent storage volumes: - db_data:/var/lib/postgresql/data # networks to join networks: - fullstack-network # GraphQL server graphql-engine: image: fullstack/graphql secrets: - postgres_password # Optional: load balance across multiple GraphQL Server deploy: # number of instances replicas: 3 restart_policy: max_attempts: 3 condition: on-failure # how the service should be updated update_config: # number of containers parallelism: 3 # time between each update delay: 10s networks: - fullstack-network ports: - \u0026#39;9002:4000\u0026#39; # Run the server after the timescale service is started depends_on: - timescale # nginx proxy: image: fullstack/lb ports: - \u0026#39;8181:80\u0026#39; depends_on: - graphql-engine deploy: placement: constraints: [node.role == manager] networks: - fullstack-network frontend: image: fullstack/frontend ports: - \u0026#39;3000:3000\u0026#39; networks: - fullstack-network # Location for permanent storage volumes: db_data: networks: graphql-network: driver: overlay attachable: true # Defines all of the secrets secrets: postgres_password: external: true Deploy all services using docker stack deploy --compose-file docker-compose-fullstack.yaml --with-registry-auth fullstack-app.\nAdd sample data to the database using this format:\ncurl \u0026#39;http://localhost:8181/graphql\u0026#39; -X POST -H \u0026#39;content-type: application/json\u0026#39; --data \u0026#39;{ \u0026#34;query\u0026#34;:\u0026#34;mutation { addTemperature(location: \\\u0026#34;London\\\u0026#34;, temperature:18.0) }\u0026#34; }\u0026#39; Check the web app at localhost:3000.\nSome notes for the docker-compose file:\nvolumes - this property will keep any data stored during the runtime of the container even if the container is removed.\nnetworks - each service will create and join their own network (the name is equivalent to the name of the service) by default. Therefore, services are not able to communicate with each other. Assigning the same network will allow them to be discoverable to other services.\nnginx\nnginx.conf\n# proxy request to a group of servers named \u0026#39;loadbalance\u0026#39; upstream loadbalance { least_conn; server graphql-engine:4000; } # pass any requests to loadbalance servers server { location / { proxy_pass http://loadbalance; } } Dockerfile\nFROM nginx RUN rm /etc/nginx/conf.d/default.conf COPY nginx.conf /etc/nginx/conf.d/default.conf add data using localhost:8181 (nginx) instead of localhost:9002 (GraphQL server) so that requests can be distributed across multiple server instances.\nCompose specification - visit Docker Docs to see the full definitions.\nThe great thing about this process is that we can now upload images for the whole application to any service providers for deployment and containers can be spun up in no time!\nMy key takeaways and thoughts from this session:\nContainers are good for automated testing, especially when you need to run end-to-end tests on multiple browsers across different operating systems which could cause significant differences in the image snapshots that are taken for your web app during the test. For example, when you use a Windows machine to generate and update snapshots and all tests run fine pre-commit. However, when the GitHub CI workflow picks up the new source code and runs in a Ubuntu container, errors will be thrown because snapshots vary in different OS. The drawback is that if you trigger the automated testing using the pre-commit hook, everyone who contributes to the project will need to either skip the tests, or need to have Docker installed. Twelve factors 1, 2, 3, 5, 10, and 11 are also good practices for non-cloud web developments. If you ever deploy a static site (built from a front-end framework/library) on GitHub Pages using GitHub Actions, you will find these factors extremely helpful. Although the 11th factor is less relevant in static sites. Automation and resilience should be key goals to any process - the application of these factors leads you to automated testing, automated deployment, auto-scaling, and monitoring. After I\u0026rsquo;ve spent some time digging into the docker world then I think I should definitely go forward and consume more sysadmin knowledge! Practical session II The second practical session focused on developing applications on AWS, we were using a virtual machine on AWS EC2 to develop and deploy a RESTful API as well as setting up CI/CD pipeline on GitLab.\nHere\u0026rsquo;s a broad summary of the setup:\nConnect the AWS EC2 virtual machine using SSH and set up AWS CLI. Clone and setup Git repository on the VM, tag the commit along the development. Build and run the API server using the Task task runner via the file Taskfile.yml. Containerised the server by creating a Dockerfile which contains multiple stages: Build* Dependencies and cache Runner Upload* images to a private repository in AWS. Config Docker to use amazon-ecr-credential-helper for the authentication of the private repository. [CI] Register GitLab runner on AWS VM. Add CI code linting* and create gitlab-ci.yml with lint and build two jobs. [CD] Add credentials of the private repository to the GitLab CI/CD page, then add the deploy job to the gitlab-ci.yml so the image will be pushed to the repository whenever a commit is made on the dev branch or when a tag is created. Create a docker-compose.yaml file which runs the image from the private repository as a service. Setup the Watchtower tool as a separate running service so the API server will be re-deployed whenever the tool finds a new update to the image. The private repository credentials are passed via a volume. Add a production service for the API server and expose it to the external world via port 80. Ta-da! An automated CI and CD pipeline is created and all we need to do is to continuously add new features to the API server. (_) indicates that these actions have been captured as tasks in the Taskfile.yml. There are other tasks such as clean and *format* which are not mentioned above.\nOne of the key takeaways of this session for myself is to automate anything I can and make use of tools like Task or GNU make. By doing so it will help you to run a series of tedious commands, integrate with any workflows that may arise in the future, and reduce the chance of human errors. I have used npm a lot and often use the package.json file for storing simple scripts. Store complicated scripts in the same file wouldn\u0026rsquo;t be ideal and sometimes cause OS compatibility issues and brings up confusion. One of the use cases of Task would be installation. When contributors fork and clone your project they wouldn\u0026rsquo;t want a long README file and have to run many commands to set up. If we can put everything in a single Taskfile and only run a minimal number of commands to get everything ready for development, much time would be saved for everyone. When it comes to large projects which may involve hundreds of microservices, such tools would make a huge difference.\nBeside the setup, there are two things in the tutorial that have drawn my attention. The first one is Semantic Versioning (SemVer) and the latter is how to write a git commit message. SemVer is a convention/scheme for version numbers and has been widely adopted in the NPM ecosystem. Although it is mainly for packages or libraries, I believe it is also useful for web applications or static websites. Separating the development history by version numbers helps to keep track of what\u0026rsquo;s been changed at any given time interval, and it is a great source of information that can turn into changelogs for stakeholders when combined with good commit messages. Over the past months, I have started to follow the Angular commit message guidelines and try to steer away from vague messages like update, fix bug, create ***. Not only have I got pull requests that are structured and easy to review, it also helped me to track past history very quickly on several occasions.\nSummary Start to adopt 12-factors methodology Sysadmin skills is essential for cloud development Containerisation and CI/CD would be a standard for projects Automate everything you can and make use of tools like Task / Make (build tools) Organise and track the project using Semantic Versioning Good commit messages make the difference! Reference Full Stack Courseware - Jim Bannister (link) Go With the Flow (link) ","permalink":"https://ylweng.com/blog/2022-03-01-twelve-factors-methodology/","summary":"\u003cp\u003eThis is my note for the event - \u0026lsquo;Building and testing apps for the Cloud\u0026rsquo; hosted by the Research Software Engineering (RSE) team at The Hartree Centre. I\u0026rsquo;m going to share what I have learned and some thought from this event, and you might find it useful and interesting if you are interested in the following:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ebest practice for building/testing cloud-native apps\u003c/li\u003e\n\u003cli\u003econtainerisation\u003c/li\u003e\n\u003cli\u003econtinuous integration, and\u003c/li\u003e\n\u003cli\u003econtinuous deployment\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"twelve-factor-methodology\"\u003eTwelve-factor methodology\u003c/h2\u003e\n\u003cp\u003eThe RSE team shared an industry-leading twelve-factor app methodology in which I have found really useful and beneficial. This methodology is specifically for designing and building software-as-a-service (SaaS) apps, but there are some factors which I think are also applicable in non-cloud environments. When I got introduced to this methodology, my immediate thought/question was: what are ideal ways to continuously deliver features without loss of efficiency, as the complexity and scale of the system increases? There is no golden approach to building a web application step by step, but this methodology is a very good place to start.\u003c/p\u003e","title":"The Twelve-factors Methodology"},{"content":"Hello and welcome to the first blog post that I\u0026rsquo;ve written for this website while I building features and designing blog styles!\nIn case if you are interested in the tech stack for this website, it is built from Angular (a Javascript framework) and Scully (a static site generator for Angular).\n","permalink":"https://ylweng.com/blog/2022-02-24-first-blog-post/","summary":"\u003cp\u003eHello and welcome to the first blog post that I\u0026rsquo;ve written for this website while I building features and designing blog styles!\u003c/p\u003e\n\u003cp\u003eIn case if you are interested in the tech stack for this website, it is built from \u003ca href=\"https://angular.io/\"\u003eAngular\u003c/a\u003e (a Javascript framework) and \u003ca href=\"https://scully.io/\"\u003eScully\u003c/a\u003e (a static site generator for Angular).\u003c/p\u003e","title":"Hello World!"},{"content":"At The University of Sheffield, I collaborate with an amazing team to develop software that supports impactful research. I enjoy turning complex challenges into practical, elegant solutions - whether it’s through clean code, smart automation, or thoughtful design. I\u0026rsquo;m especially driven by efficiency and love finding ways to streamline workflows. If you’re into building better tools or just enjoy solving interesting problems, I’d love to connect.\nProfessional experience My career began as a Research Systems Developer, where I focused on building and maintaining systems to support research activities. Over time, I transitioned into the role of a Research Software Engineer, expanding my expertise to design and develop robust software solutions tailored to the unique challenges of academic and scientific research. Some of my key experiences include:\nFull-Stack Development JavaScript/TypeScript Ecosystem Python Ecosystem Agile Methodologies Software Lifecycle Management Version Control Containerisation and Orchestration Test-Driven and Behaviour-Driven Development CI/CD Pipelines System Design Data Visualisation \u0026hellip; and more! Academic background I hold a bachelor’s degree in Discrete Mathematics (Mathematics and Computer Science). For my final-year project, I developed and simulated an operational research model to optimize flight rescheduling for an airport modelled after London Heathrow. Fascinated by the intersection of statistics and machine learning, I pursued a master’s degree in Statistics, where I designed a configurable data-analytics pipeline in R. This pipeline transformed raw NBA performance data into insights on player and team abilities.\nBeyond work Outside of work, I enjoy immersing myself in:\nCooking Reading Music Strategic and simulation games Exploring AWS Kubernetes DevOps Watching WebAssembly Rust Go About this website This website serves multiple purposes:\nExploration: A fun way to dive deep into various front-end technologies. Knowledge Sharing: Sharing my thoughts and experiences in the hope they help others achieve their goals. Resource Hub: A place to store valuable resources and notes. Experimentation: A sandbox for experimenting with data visualization and other tools. And More: Continuously evolving to meet new needs and ideas. Built with Hugo GitLab Docker Visual Studio Code Cloudflare Workers ","permalink":"https://ylweng.com/about/","summary":"\u003cp\u003eAt The University of Sheffield, I collaborate with an amazing team to develop software that supports impactful research. I enjoy turning complex challenges into practical, elegant solutions - whether it’s through clean code, smart automation, or thoughtful design. I\u0026rsquo;m especially driven by efficiency and love finding ways to streamline workflows. If you’re into building better tools or just enjoy solving interesting problems, I’d love to connect.\u003c/p\u003e\n\u003ch2 id=\"professional-experience\"\u003eProfessional experience\u003c/h2\u003e\n\u003cp\u003eMy career began as a Research Systems Developer, where I focused on building and maintaining systems to support research activities. Over time, I transitioned into the role of a Research Software Engineer, expanding my expertise to design and develop robust software solutions tailored to the unique challenges of academic and scientific research. Some of my key experiences include:\u003c/p\u003e","title":"About"}]