Many universities are starting to specifically offer programs in data analysis, and Data Science, as a popular application major in recent years, has become increasingly competitive. However, this very popularity has led to a bandwagon effect where many students lack a clear self-positioning and decide to apply simply because the major is hot. Such a hasty decision may put you in a very awkward position.
So I invited a highly capable friend, Teacher Han, to give everyone a popular science introduction to what kind of people are truly suited for studying DS, and what kind of development they can expect after studying DS.
Q&A
Q: “Why Data Science? With a double degree in Mathematics and Physics, such an impressive academic background, why didn't you continue studying in Math or Physics? What opportunity sparked your interest in Data Science?”
Mathematics and Physics are both foundational disciplines that provide a theoretical basis for practical applications. They can serve as foundational disciplines because many fields require this knowledge as a foundation; conversely, these disciplines offer us very rich options for future development, which is a major reason I chose to study Mathematics and Physics in the first place.
Through my undergraduate studies, I found my interest in applied fields outweighed my passion for theoretical science. So I shifted my focus toward applied areas, using applied mathematics as an entry point to find a specific direction of interest within it.
Applied mathematics mainly divides into three areas: computational mathematics, financial mathematics, and statistics. Computational mathematics leans relatively theoretical, finance wasn't my forte, so that left the statistics direction. That was just when DS was emerging, and academia widely believed this would be the main direction for future development. NYU had also just established its DS program. So this interdisciplinary field integrating statistics, math, and computer science caught my eye, and by chance, I caught this wave of academic trends.
Q: Based on your understanding, what does Data Science actually involve studying? How is it different from Statistics?
In my understanding, DS is a combination of Statistics, Mathematics, and Computer Science.
From the statistics perspective, DS mainly involves probability distributions, statistical inference, and linear regression;
From the mathematics perspective, DS primarily involves linear algebra;
From the computer science perspective, DS mainly involves programming and algorithms.
Simply put, DS shows us how many ways there are to process experimental data and what conclusions and implications each method corresponds to.
On the surface, DS and Statistics share many, many commonalities. For example, both require studying probability and statistical inference. However, when broken down, there are some differences.
First, Statistics places greater emphasis on theoretical foundations than DS. For example, hypothesis testing in statistical inference is a key and challenging point in Statistics, whereas DS only needs to know its manifestation form and calculation steps to suffice;
Second, Statistics not only needs to know how to analyze data using hypothesis testing but also how to design experiments to obtain the desired data results;
In addition, Statistics does not require algorithms to compute data; DS extends statistical methods to create many algorithms for calculating relationships between data;
Moreover, Statistics already has many established tools that make direct statistical processing convenient, while the diversity of algorithms in DS requires using programming languages to implement them on your own.
Q: During your graduate studies at NYU, what skills can be cultivated and enhanced?
First, all professors in my courses at NYU uniformly assumed that students had already mastered the required programming skills. All DS-related courses, whether offered by the Computer Science department or not, taught only theoretical knowledge in class. How to put it into practice was left for students to explore on their own through homework assignments.
I remember the first assignment required using Python. At that time, I knew nothing about Python. I spent two full days self-learning it and directly used the assignment for hands-on practice. Second, the sheer volume of coursework left no time to breathe, especially in the first two weeks after just learning Python. Although a week was given for each assignment, for me it was only barely completed on time with the help of classmates and TAs. Yet, on the day of submission, a new wave of assignments would come at me. It was precisely this compact practical process that kept me constantly reinforcing every knowledge point learned, while my programming ability improved rapidly through continuous practice.
Worth mentioning is that all assignments had a system, progressing layer by layer around one problem, ultimately arriving at a global conclusion at the last question. This system helped integrate learned theories in the order of practical operations. Repeated practice like this many times cultivated a sensitivity to various types of data. The next time a problem is encountered, I can clearly dissect the corresponding thought process and algorithm from beginning to end. This, I think, is a very important ability.
Many students interested in the data field want to enter this area but don't know where to start studying. Any advice for novice-level rookies?
Still, consider things from the three directions of Statistics, Mathematics, and Computer Science. For Statistics, in my opinion, basic statistics is a theoretical system built upon discrete and continuous probability distribution models. Therefore, for beginners, thoroughly mastering these probability distribution models will be very helpful for deeper study later. Next is linear regression.
Now about Mathematics. The mathematical concepts used in DS, in my personal view, are broader than those in Statistics. Here I mean mathematical concepts rather than mathematical definitions. That is to say, if you want to make certain breakthroughs in this field, generally speaking, the definitions proposed in linear algebra, such as vectors, matrices, spaces, etc., convey a very strong mode of thinking that 'algebracizes' problems during study, which is also crucial for DS.
Finally, let's discuss the computer science aspect. For this area I would recommend directly learning Python. If you have a background in systematic computer science study, that will be a great help for learning new languages. If you have no prior programming experience, Python can also be a good choice. On one hand, Python is open-source, so there are tons of resources for reference; on the other hand, many DS-related libraries are already quite mature. Furthermore, initially I don't recommend preparing algorithms from the perspective of a computer science discipline. Based on my own learning experience, it might be better to start preparing from a mathematical or statistical perspective, and after building a certain foundation, then get involved with some algorithm concepts related to computer science.
One more thing: for those who already have a background in the above three areas, I think it’s very important to deeply understand the concept of parallel computing. Because single-machine processing of big data is gradually becoming unrealistic, more powerful computing resources are needed. But for those with a significant gap in background, this matter can be put on the agenda a bit later.
Q: Any suggestions for students regarding applications? Do you recommend any school programs?
Possess programming ability before enrollment—of course this varies depending on each person's background. Python and R are fundamental programming languages for data science. On top of that, mastering other programming languages, such as SQL, MATLAB, and C++, can all be written into your resume as programming skills, which adds icing on the cake.
Having relevant internship experience is best. If not, do some related projects. You can seek help from professors at your current school and look among them for opportunities to join such projects. These practical experiences not only benefit applications but also help you adapt more quickly to study and research in this field.
Browse DS-related forums or groups to understand development trends, which may help with your personal statement. How much help this provides is hard to quantify, but after all, since you need to enter this industry, understanding the industry forefront and dynamics is an excellent preparatory course. Applications have their own procedures and criteria. As I understand it, the most fundamental reason an applicant is admitted is that they meet the school's expectations and requirements; the school believes you have great passion for this field and are capable of handling the coursework. Of course, there will be some quantitative requirements upon admission. The above three points can not only strengthen your application but, more importantly in my opinion, setting aside applications, they will also contribute to your own knowledge base.
Q: Having been in New York for so many years, tell us about your life experience.
Relative to administrative divisions, New York is more of a regional concept to me. I lived in Stony Brook, on the north shore of central Long Island, for about 5 years; in Queens for 1 year; in New Jersey for 1 year; and studied in Manhattan for two years. New York is pretty much the general term for my daily living area.
When I was at NYU, the academic buildings were all downtown, in the bustling city center, with very, very many delicious restaurants—so much so that I still don't know where the NYU campus cafeteria is. What I ate most was still McDonald's, the one right next to the QR line. Usually at night, I'd buy a meal and take it on the subway home. I couldn't eat the good food because there were piles of assignments, and every day was a deadline—no exaggeration. The subway ride would take about 80 minutes; after moving to New Jersey it became a bit closer. The subway was the most relaxing time, when I felt I didn't have to face never-ending classes and assignments and the same chicken with more batter than meat again. At that time, if the train was on time and I got a seat, that was the luckiest thing that day.
Probably about three or four times a month, I would sunbathe in the garden enclosed by the buildings of Courant and Stern. Only at these moments would I occasionally think about experiences or feelings or reflections of that kind. For me, this was the truest feeling of living in New York. Perhaps the time to stop and reflect was miniscule, but I could feel that I was genuinely doing something meaningful for myself. But it's very hard for me to say what that meaning is. But later, when I left Manhattan and returned to Stony Brook, I sometimes still missed that place. After all, the Vietnamese beef noodle soup next door had plenty of meat and rich broth and was really tasty.
What is DS?
Data Science is the extraction of information and knowledge from data. It is an extension of data mining and predictive analytics and also a process of discovering knowledge and data. So, in layman's terms, Data Science is about uncovering potential information within data through analysis.
Using vast amounts of data for analysis to support business decision-making (data-driven decision making) is the ultimate purpose of data science. Broadly speaking, the research and application directions of Data Science fall into the following parts:
Predictive Analytics:
Analyzing data to predict possible future events.
Descriptive Analytics:
Analyzing data to identify characteristics of past events and trends in ongoing events.
Prescriptive Analytics:
Analyzing data to find the best measures and achieve optimal results.
Data Science-related skills can roughly be divided into three categories:
Algorithms: Theoretical knowledge in mathematics, statistics, algorithms, and models is the core of solving any data problem. The algorithm models behind data science and how data experiments are designed and tested are indispensable skills.
Engineering: The acquisition, processing, and storage of data are essential knowledge. At the same time, proficiency in programming languages and distributed computing will help you handle large-scale data.
Communication: Data is like a foreign language to laypeople. How to transform data analysis results into highly readable business reports is a skill that must be mastered.
DS (Data Science) vs. DM (Data Mining)
Data Mining is about finding trends in datasets and using these trends to determine future patterns. It is an important step in the knowledge discovery process and usually involves analyzing large amounts of previously overlooked historical data.
Data Science is a field of study that includes everything from big data analytics, data mining, predictive modeling, data visualization, mathematics, and statistics. Data Science is referred to as the fourth paradigm of science (the other three being theory, empiricism, and computation). Academia often conducts exclusive research in the field of data science.
The following are the differences between Data Science and Data Mining:
Data Mining is an activity that is part of the broader Knowledge Discovery in Databases (KDD) process, while Data Science is a field of study within applied mathematics or computer science.
Generally, Data Science is broad, while Data Mining is considered niche.
Some activities under Data Mining, such as statistical analysis, writing data flows, and pattern recognition, can intersect with Data Science. Therefore, Data Mining becomes a subset of Data Science.
Machine learning in Data Mining is more used for pattern recognition, while in Data Science it has broader applications.
Note: Data Science and Data Mining should not be confused with Big Data analytics, and both miners and scientists can work with large datasets.
Curriculum of the Data Science Major
American universities roughly fall into four categories:
Research Schools, which must have a PhD program.
Comprehensive Teaching Schools, which may not have a PhD program and primarily focus on teaching, aiming to prepare students for work as quickly as possible. However, they may also have research-oriented Master's degrees.
Four-year Liberal Arts Colleges, which are less research-intensive than comprehensive universities, having virtually no research and essentially only undergraduate education.
Two-year Community Colleges, which are merely extensions of high school.
Data Science Specialized Courses
For some schools, the Data Science major selects some Data Science-related content from three different disciplines as core courses. Besides these, there are several courses beginning with the name Data Science, for example:
Introduction to Data Science — teaching what Data Science is and what it can do
Data Analysis
Data Mining
Big Data — explaining how to use free distributed systems like Hadoop and Spark to process large amounts of data in parallel
Capstone — graduation internship or graduation design project
Academic Preparation for Data Science
1. Courses:
• Statistical Theory
• Data Mining / Machine Learning
• Optimization
• Data Visualization
2. Computer / Programming Languages:
• R/SAS
• Database SQL
• Object-Oriented Programming Python
• Big Data Engine MapReduce/Hadoop
Popular Master’s Programs in Data Science
Columbia University
Columbia University's MS in Data Science program is among the most competitive first-tier Data Science master's programs in the U.S. Columbia has a world-class Institute for Data Sciences and Engineering where students can participate in experiments and research projects. This program was newly launched in fall 2014, showing the university's emphasis on this field.
The program is housed under the Engineering School and focuses on data mining, algorithms, and statistical modeling (e.g., Algorithms for Data Science, Machine Learning for Data Science, Statistical Inference & Modeling). Its curriculum is designed to meet industry needs. The program lasts one year, requiring 30 credits, with no thesis requirement. It offers four required foundational courses that lead to professional achievement certification, allowing students to apply data science technologies to their fields of interest on this foundation.
Applicants are expected to have some mathematical and programming foundation, preferably having taken courses in calculus, linear algebra, computer programming, etc. No mandatory work experience is required, but it can be a plus. GRE scores are required; TOEFL 100, IELTS 7.0. GMAT cannot substitute for GRE.
Students in the program have opportunities to engage in original research, including capstone projects, and interact with industry partners and faculty. Graduates can pursue careers in finance services or tech-oriented IT companies.
Below are the main courses:
Of these, the seven required courses focus heavily on both Computer Science and Statistics. The curriculum has sufficient difficulty and value. The Capstone Project also offers excellent practical and application opportunities. Coupled with Columbia's Ivy League prestige and the overall teaching quality and advantageous resources of the Engineering School, the program's competitiveness is very high.
Application requirements:
As with most DS programs, there are certain requirements for the applicant's background in CS and Math. Basic computer language and math foundational courses must be completed.
Duke University
Duke’s Master in Interdisciplinary Data Science (MIDS) program is a 2-year program designed to cultivate a new generation of leaders who can use computational strategies to spark innovation and insight and think quantitatively adept. It aims to train students to become data scientists who can contribute to any field, encouraging better data utilization through interdisciplinary training and team science experiments. The program is small, admitting 25-35 students per year.
Regarding application requirements, a background in mathematics or computer science is not mandatory, but coursework in calculus, linear algebra, statistics, etc., is preferred. GRE scores are required; TOEFL 90, IELTS 7. Graduate outcomes are good, with abundant internship and job opportunities, commonly in fields like computer science, finance, and biological sciences.
University of Pennsylvania
Penn’s Master’s program in Data Science lasts one and a half to two years, organically combining cutting-edge courses on core topics such as machine learning, big data analytics, and statistics with diverse electives, giving students the opportunity to choose to apply their skills within specific targeted professional areas.
The program includes foundational courses and an internship in the applied field of data science. GRE is required, with no minimum score requirement. TOEFL 100, IELTS 7.5. Graduates have positive employment prospects and a high employment rate, entering fields such as engineering technology, consulting, and decision-making.
Northwestern University
Northwestern’s Master of Science in Analytics (MSiA) program, offered by the Department of Industrial Engineering and Management Sciences, lasts 15 months. Core courses consist of statistics, machine learning, optimization, databases, and data processing, each building on the previous one, emphasizing academic research in practical business contexts.
Application requirements include GRE scores; TOEFL 95, IELTS 7.5. It suits students with undergraduate backgrounds in engineering, business, computer science, mathematics, or information science and technology. The program is business-oriented and relatively short, providing many internship opportunities. Most graduates enter directly into work, often in the financial sector.
Cornell University
Cornell’s Master of Engineering program in Operations Research and Information Engineering (Data Analytics concentration) lasts one year and totals 30 credits. Applicants should have undergraduate backgrounds in mathematics, engineering, physics, chemistry, or mathematical economics, and have completed courses in calculus, probability, statistics, computer programming, and data structures.
About 50 students are admitted each year, with about 30 being Chinese. This program offers great flexibility in course selection and priority for enrolling in Computer Science department courses. Whether you want to study CS, IS, business, finance, statistics, you can choose them all. This degree of freedom in course selection is rare across the nation.
Applicants need TOEFL 100 (writing no less than 20, listening no less than 15, reading no less than 20, speaking no less than 22); IELTS 7; GRE math minimum 166, reading minimum 165, writing minimum 3.5; GMAT not accepted. The average starting salary for 2015 graduates reached $83,925, with most entering internet companies like Apple, Microsoft, Oracle, and Facebook.
How to Strengthen Your Background?
Some students might ask, “My background is not that strong or not a great match, what can I do to become more competitive?”
1. Research
For research, it's best to find quantitative-related projects. If you really can't, you can use relevant course projects (course projects). If even that is not possible, CS-related ones can work. But remember, lacking any research experience would be a significant weakness.
2. Competitions
There are many competition platforms, such as the recently popular Kaggle, and also Alibaba’s Tianchi, SODA, WID, Datahex, etc. These are all platforms with data competitions.
3. Internships
For internships, the optimal choice would be a data role at a data company. However, the reality is that such positions, due to their critical importance, rarely hire interns. So it is recommended to find internships related to statistics/quantitative analysis or computer science.
Career Directions in Data Science
1. Machine Learning Engineer
Represents a direction with higher technical content. The work mainly involves developing machine learning systems and using these systems to solve practical problems. Generally requires shipping production code; the output is a data product.
2. Data Analyst
The work content is commonly known as analytics (product analytics or business analytics), extracting insights from data, estimating ROI, providing direction for products. The tools used are generally more basic, such as writing SQL queries to fetch data, doing simple analysis with R/Python, and plotting with Tableau/Excel. Being able to develop dashboards on your own puts you among the technically stronger analysts. Work requires producing various reports; at the statistical level, understanding basic t-tests and linear regression is sufficient.
3. Data Scientist
Many people say, “I want to be a data scientist, I want to do machine learning,” and this type of position is exactly what they imagine.
The work content primarily involves advanced modeling, designing technical solutions for complex problems, such as Uber’s ETA for pickups, various pricing systems, fraud detection in Airbnb and finance, Amazon logistics management, social networks at Facebook/LinkedIn, or experiments on marketplace scale for two-sided markets like eBay, Airbnb, or Uber. These examples sound like they can’t be solved just by writing SQL, nor can they be done just by knowing how to code; they all require relatively deep domain knowledge.
4. Business Analyst
Considered a type of data analyst, responsibilities mainly include collecting, processing, executing, performing statistical analysis on data, writing business analysis reports, data visualization, and storytelling. Requires mastering Excel, PowerPoint, Python, SQL, etc., and may require visualization tools like Tableau. Such jobs often demand high proficiency in the company’s business domain knowledge.
Which Companies Can You Choose When Job Hunting?
Various industries are using big data to solve problems, so many companies have relevant positions.
Tech companies: Twitter, Microsoft, BAT, Sogou, Meituan, Didi
Financial companies: Capital One (which has been using data methods for risk control and more since its founding), Goldman Sachs, BlackRock, Hedge Funds
Consulting firms: Booz Allen (helping U.S. government departments with counter-terrorism and vetting), McKinsey, IBM (traditional consulting firms are also gradually shifting toward data)
Companies specializing in data analytics platforms: Cloudera, Hortonworks, Databricks
Other companies: Verizon (telecom), Comcast (media), consumer goods companies, medical companies
Enterprise Demand for Data Science
Why emphasize Data Wrangling? Let’s look at it in light of companies’ demand for Data Science talent.
If a person has a PhD title and is a Machine Learning expert, then in the company they generally specialize in doing Machine Learning algorithms, thinking about parameters, models, etc., every day.
However, people who have actually done related work will tell you that 80% of the time in Machine Learning is spent on Data Wrangling.
So now many companies prefer to hire people specifically to handle this aspect of data preprocessing — extracting data from complex sources, taking out the needed parts to maintain, and then transforming the data into a form usable for Data Modeling. This part of the work is extremely useful.
Companies will not hire an undergraduate to do Machine Learning. But using a PhD for Data Wrangling is too expensive.
At the undergraduate level, one is fully capable of handling Data Cleaning, Data Processing, and some simple Data Analysis work. For those who want to start Data Science upon graduating with a bachelor’s, Data Wrangling and Data Analysis are very suitable directions.
What Resources Help Our Learning?
01 Free Datasets
The first recommendation is Kaggle, currently the largest communication platform for data scientists, now part of Google.
Kaggle provides three things:
Data: People share free, open, public data.
Competitions: Many companies set up various competitions: set a goal, everyone competes to see whose result is better.
Solutions: After completing data analyses, many people are willing to share their solutions; this part is called kernels.
Data Science, Machine Learning, or NLP has thrived thanks to an excellent tradition in the computer field: Competition.
Someone provides a standard dataset; everyone tries various methods on it; then compares results to see whose effect is better and whose can still be improved.
Kaggle continues this tradition, being a platform specifically aimed at Data Scientists to promote the development of Data Science.
The second recommendation is KDD-CUP: Knowledge Discovery and Data Mining.
This platform has a longer history. Its earliest user base was database people. It also hosts useful resources, including data, competitions, tool introductions, and method sharing, but overall it’s more database-oriented.
The third recommendation is Kdnuggets.
This site leans more toward Business Analytics, Big Data, Data Mining, Data Science, and Machine Learning.
Amazon’s AWS also provides free public datasets, but these are generally large datasets.
02 Online Educational Resources
If you are interested in online courses, I recommend Coursera. It has series courses from UIUC, UMichigan, Johns Hopkins University, etc.
UCSD has a Big Data series which I personally highly recommend.
A similar course platform is Udemy. It’s similar to Coursera but primarily focused on individual courses. Unlike Coursera, which often offers series courses allowing users to systematically complete four or five courses to learn a relatively complete knowledge set, Udemy requires you to search for relevant courses on your own.
Another platform is Datacamp, which specializes in Data Science courses. It has videos, hands-on programming, and R, Python, and Statistics.
For undergraduate-level students who need specialized training in a programming language to have a very handy tool they can wield for anything, Datacamp is an excellent choice. The course format there usually involves some explanation, then some examples where you directly follow along and do. For going deeper, Datacamp is slightly lacking.
