Introduction​

Using web crawling programs to extract list web pages is a one of those common web data extraction tasks. For engineers to write web crawlers, how to efficiently code and generate extraction rules is quite necessary, otherwise most of the time can be wasted on writing CSS selectors and XPath data extraction rules of web crawling programs. In light of this issue, this article will introduce an example of using open source tool Webspot to automatically recognize and extract data of list web pages.

Webspot​

Webspot is an open source project aimed at automating web page data extraction. Currently, it supports recognition and crawling rules extraction of list pages and pagination. In addition, it provides a web UI interface for users to visually view the identified results, and allows developers to use APIs to obtain recognition results.

Installation of Webspot is quite easy, you can refer to the official documentation for the installation tutorial with Docker and Docker Compose. Execute the commands below to install and start Webspot.

# clone git repo
git clone https://github.com/crawlab-team/webspot

# start docker containers
docker-compose up -d

Introduction​

"If I had asked people what they wanted, they would have said faster horses" -- Henry Ford

Today is the era of artificial intelligence. Whether it is ChatGPT or the various intelligent applications that follow, many people see the upcoming sci-fi world that was almost unimaginable a few years ago. However, in the field of reptiles, artificial intelligence does not seem to be involved too much. It is true that crawlers, as an "ancient" technology, have created many technical industries such as search engines, news aggregation, and data analysis in the past 20 years, but we have not seen obvious technological breakthroughs yet: crawler engineers still mainly rely on technologies such as XPath and reverse engineering to automatically obtain web data. However, with the development of artificial intelligence and machine learning, crawler technology can theoretically achieve "automatic driving". This article will introduce the current status and possible future development direction of the so-called intelligent crawler (intelligent, automated data extraction crawler technology) from multiple perspectives.

Current Web Crawling Technology​

A web crawler is an automated program used to obtain data from the Internet or other computer networks. They usually use automated scraping techniques to automatically visit the website and collect, parse and store information on the website. This information can be structured or unstructured data.

Crawler technology in the traditional sense mainly includes the following modules or systems:

1. Network request : initiate an HTTP request to a website or web page to obtain data such as HTML;
2. Web page parsing : parse HTML to form a structured tree structure, and obtain target data through XPath or CSS Selector;
3. Data storage : store the parsed structured data, which can be in the form of a database or a file;
4. URL management : manage the URL list to be crawled and the URL list that has been crawled, such as URL resolution and request for paging or list pages.

Introduction​

"Entities should not be multiplied without necessity" -- Ockham's Razor Principle

Graph Theory is a mathematical foundational theory that has been severely underestimated by the public. It doesn't study images, pictures, or charts, but rather an abstract and simple mathematical theory. The graph in graph theory is an abstract concept, very similar to a relationship network, with corresponding nodes (or vertices), and associative relationships or edges between nodes. Graph theory concepts are very simple: graphs, nodes, and edges. This article will briefly introduce basic concepts of graph theory and its applications in the real world. (Note! This is not a scientific paper, so there won't be boring mathematical formulas - please enjoy reading)

Graph Theory Overview​

In graph theory, there are three important concepts:

1. Node: Can be understood as an entity, such as Zhang San, Li Si, Wang Wu in a relationship network;
2. Edge: Can be understood as relationships between entities, for example, Zhang San and Li Si are husband and wife, Wang Wu is their son;
3. Graph: Can be understood as the collection of all nodes and edges, such as the happy family composed of Zhang San, Li Si, and Wang Wu.

From these three basic concepts, we can infer relationships between nodes. For example, Li Si's older brother Li Yi would be Wang Wu's uncle, and Wang Wu would be his nephew.

Introduction​

Data Analytics and Business Intelligence are important business modules for many enterprises to implement digital strategies. We previously introduced indispensable parts of the data field in 《浅谈数据：数据领域需要掌握些什么？》, namely software tool auxiliary services in architecture and processes. The open source data analysis platform Apache Superset introduced in this article can provide such services. This article will briefly introduce how to install, deploy, and use Superset.

Superset Introduction​

Superset is an open source self-service data analytics platform incubated by the Apache Foundation. It can be seen as an open source version of Power BI or Tableau, though Superset's interactive interface is limited to Web. The entire system is based on Python Flask and integrates with mainstream relational databases like MySQL, Postgres, SQL Server, as well as modern databases like ElasticSearch, ClickHouse, Snowflake. The frontend visualization analysis interface is very similar to Power BI and Tableau, with relatively simple operations. Therefore, if you need to build an enterprise-level data analytics platform like Power BI or Tableau without spending money, Superset is an excellent choice.

Introduction​

Complexity is an eternal challenge in software engineering. As project scale grows, complexity increases exponentially, and if left uncontrolled, it can ultimately lead to project failure.

In the world of software development, complexity is everywhere. From simple "Hello World" programs to large-scale distributed systems, complexity always accompanies our development process. As software architects and technical leaders, understanding the nature of complexity, its sources, and how to manage it is a core skill we must master.

Introduction​

"Architecture is about the important stuff... whatever it is." -- Ralph Johnson

Architect is a position with powerfulness and respect. When you hear that someone is an architect of a company, will you feel a sense of awe? Architects are usually believed to be relevant to system design, technical strength, leadership and influence. It is precisely for this reason that many of the positions of architects in the enterprise are held by experienced and skilled senior software engineers. However, the definition of an architect in the software industry is not quite clear: cloud service providers such as Amazon and Alibaba Cloud have their own teams of architects, but most of them provide after-sales service to customers under the name "architect"; Some architects are nothing more than using his rich experience and excellent strength to solve technical problems, which is equivalent to what a senior software engineer can do. These are very different from the omniscient architect who designs the architecture diagram.

The concepts related to architects in this article mainly come from a book Fundamentals of Software Architecture (authored by Mark Richards, Neal Ford) that I have been reading recently. This article will briefly introduce about what a pragmatic architect should do and required skills.

Chief Engineer​

First of all, an architect should be a chief engineer of the entire software project, who is responsible for the overall design, implementation and quality of the software project. Therefore, for software architects, they need to have excellent programming skills, good understanding of the software project development process, and a certain breadth and depth in various technical areas. Not only that, because the architect is the chief person in charge of the technology side, he (or she) usually needs to think it through, from a perspective of the system as a whole, about how the various modules interact, whether the division of functional services is reasonable, where the bottleneck of the whole system will be, and so on. These are all in the technical area.

Introduction​

There are many excellent open-source projects on GitHub, where the code is transparent and available to everyone. As software developer, we can learn a lot from them including software engineering, unit testing, coding style standardization, etc. We can even find issues by looking into their code, and submit pull requests to contribute to tech communities. Today we are going to dig into the source code of a popular Golang open-source project on GitHub, robfig/cron, which is small and with clear annotations, and is very suitable for new developers to learn how to read and analyze source code.

Environment Preparation​

First, we are going to fork the project into our personal repo. We can do so by clicking the Fork button and enter the project name. After it's forked, we can either download to our local, or click Create codespace on master to create a Codespace on the GitHub repo's home page. Codespace is a Azure-based GitHub service that allows developers to develop remotely, and it is now available to individuals. Let's give it a try!

After clicking, a new page will be opened in the browser, where there is a web interface of VS Code, displaying the directory, code and terminal of the project. See the figure below.

Introduction​

"All models are wrong, but some are useful."--George Box

For many people, data analysis is familiar but inaccessible. Data rookies think that fancy dashboards look cool. Operational managers think that time-series charts can help them make business decisions. Programmers think that data analysis is nothing more than fetching the data of target fields from the database according to certain requirements. These views are all correct, but incomplete. The really useful data analysis is not only to present charts and numbers, but also to fully combine data insights with the business knowledge, which is meaningful to add values to the business. Understanding some basic statistics knowledge is helpful to discover insights.

Unreliable Averages​

We can often see that many data reports display daily, weekly, or monthly averages, such as the daily average sales of the current month, the monthly average number of visits last year, and so on. The statistics of the average value will be helpful for some specific situations, such as the time of getting up every morning, and the offset of hitting the target. But more often, you are likely to be skeptical about the average value, because it fluctuates up and down quite a lot, and the fluctuation range is not small. The root cause here comes from the Non Linear Distribution in the real world. The distribution of websites' response time, number of web page visits, and stock trend is non-linear. In these non-linear distributions, the average value fails because there are a large number of outliers that cause the average value to be seriously skewed. As shown in the figure below, for normal distribution or gaussian distribution, it is linear, so the average value is at the peak in the middle; But for gamma distribution, given that it is a nonlinear, its average value seriously deviates from its peak, and when there are more outliers, its average value will further deviate from its central position.

Therefore, for these non-linear distributions, the average value is not a reasonable indicator, but we can use median instead to describe its overall distribution. There are many tools to deal with non-linear distributions, one of which is the Box Plot. As shown in the figure below, the two distributions are abstracted into a box and several lines, where the box center line is the median, and the edges are first-quartile and third-quartile lines. In this way, it is not necessary to do too much complicated analysis to quickly get a general idea about the distribution.

Introduction​

"Organisms are living upon negentropy."--What is Life? --by Erwin Schrödinger

In today's Internet era, data is an important asset for enterprises. We generate data all the time: every time we open a mobile app, place an order online, or even drive through traffic lights, data will be generated. Data is everywhere. This is especially true in enterprises. With so much raw data and increasingly mature data analysis technology, entrepreneurs are excited, because this is the gold mine piled up for companies. However, things are not as simple as imagined. It is not easy to extract valuable treasures from these messy so-called "gold mines" that look like garbage dumps. In my previous article Talking Data: What do we need for engaging data analytics?, the concept of Data Governance was mentioned. This article will introduce how data governance creates value in the chaos of enterprise data from the perspective of enterprise data management.

Isolated Data Island​

Large and medium-sized enterprises (generally more than 100 people with multiple departments) will encounter the problem of management chaos while their business grows fast. The sales department has its own sales statistics, typically in the form of large and scattered Excel files; The IT department manages the inventory systems by themselves; The HR department maintains an entire personnel statistics list. This situation will lead to many headaches: the boss often complains that he or she can only receive management reports every week; Managers looked at the up and down data in the report, doubting the data integrity; employees work overtime to sort out the data for the reports, but they are often questioned about data quality. Sounds familiar? These are very common issues in companies. The direct cause is the so-called Isolated Data Island problem.

The main reason for isolated data island issues is that data from various departments or teams is disconnected. Many times, because of the rapid growth of business, some teams need to quickly set up a data system, but are incapable of developing it in time, so they can only use Excel or some fast and effective data entry tools to ensure the efficiency of business operations. With the growth of business, these hacking workarounds continue to derive new internal processes, and bottlenecks will gradually emerge after reaching a certain scale, especially when integration with other departments or external systems is required.

Introduction​

Nowadays, many cloud-native and distributed systems such as Kubernetes are written in Go. This is because Go natively supports not only asynchronous programming but also static typing to ensure system stability. My open-source project Crawlab, a web crawler management platform, has applied distributed architecture. This article will introduce about how to design and implement a simple distributed system.

Ideas​

Before we start to code, we need to think about what we need to implement.

• Master Node: A central control system, similar to a troop commander to issue orders
• Worker Node: Executors, similar to soldiers to execute tasks

Apart from the concepts above, we would need to implement some simple functionalities.