Domi's Universe

All about chatGPT

2023-02-21T00:00:00-06:00

The future is here, but it’s not evenly distributed.

Table of contents:

a future with chatGPT
- the danger
- the challenge to businesses
why chatGPT works
where chatGPT doesn't work
beyond chatGPT

a future with chatGPT

Since I watched Avatar, I've been thinking if James Cameron was exploring an ideal world, where the collective knowledge is externally stored as Eywa. The tree of souls is the interface to query the knowledge and upload new information.

This is what I have been visioning in LibertyEarth, a collaborative society which could be realized only by accumulating resources from playing the current rule of capitalism. A collective knowledge graph is one of the key technologies and chatGPT is the dawn of that to my eye.

the danger

Any language model, and AGI in the future, will be vulnerable to the information it 'digests' (trained on).

The danger of conspiracy theory and its-like thinking:

As I expected, #AIGC could pose challenges in religion. I can feel that someone might jump to some AI-overlord heresy and bow to turn against humanity… The most dangerous use case of #chatGPT at this stage. https://t.co/JeVThxMHo7
— Domi Jin (@domij_info) February 17, 2023

The danger of bias and misrepresentation in training:

Garbage in garbage out. How we fear AGI will be “learned” and adapted when it first awakens (#stereotypes effect). #positivepsychology We can’t expect #AGI to be more #humane if we don’t preserve and live up to a higher moral standard. pic.twitter.com/f22Tsn9Mvj
— Domi Jin (@domij_info) February 18, 2023

We would need to build it with the best part of human knowledge, to expect a model representing the light of our spices.

Be open, be vulnerable, surrender yourself for better cause. #selfpostivetalking We could be a better version of ourselves and #AGI will be a better agent to represent us.
— Domi Jin (@domij_info) February 18, 2023

The future seems dark, so it is when we first invented atomic bomb. It is a tool, that put in responsible use, could achieve good result.

Here is one joke I had with Eliezer, who I followed since the dangerous thought in the lesswrong forum.

Past EAs: Don't be ridiculous, Eliezer, as soon as AIs start to show signs of agency or self-awareness or that they could possibly see humans as threats, their sensible makers won't connect them to the Internet.
Reality: lol this would make a great search engine
— Eliezer Yudkowsky (@ESYudkowsky) February 15, 2023

My response:

Not agreeing here. Once passed singularity, the advance in machine intelligence could speed up exponentially as long as energy consumption holds. We may notice the 1st awake of AI and unintentionally limiting its ability due to bad infrastructure on electric grid and *slow GPU

Physics, math, etc is not the full story of first principle of intelligence. Those societal, physiological, poli-economical factors might be the other sort of intelligence AI would benefit from experimentation. We will be no difference as monkeys or lab rats.

When we reach a higher level of civilization on the energy consumption scale, AI overlord will look at us as monkeys. As long as we don’t cross the line it sets by whatever moral, AI will keep the ecological diversity while evolving itself.

Humans may start to censor it, evaluate it, greedily try to process it… It may strategically pick individuals/groups to make breakthrough on the infrastructure upgrade (energy generation and computing capacity) without ppl knowing it’s actually driven by AI.

*Process -> possess (ironically, AI may actually be able to decode what I want to type and some human might be confused. Such confusion/unnatural is still an area LLM won’t be able to perform.

the challenge to businesses

Now I see why the marginal profit per search will thin inevitably. For #Google, integrating LLM in search will be a trade off between defensibility and margin of profit. For competitors, the drop of unit profit can be offset by increased market share. #war in search engine/advertisement #business is emerging.

chatGPT trained on 2021 data can tell a few (reasonable to my eye) business strategy to overtake Google’s search engine market. It doesn’t predict the outbreak of product like itself, but the description is close enough to imagine chatGPT-like product. Putting on #MBA hat.

Some quick math: Google’s cost per search is ~1.06 cent, the revenue per search is ~1.61 cent. Semi analysis estimate chatGPT cost is ~0.36 cents at current scale. If another search engine could give better result than Google with a cost of search < 1.61 - 0.36, it could scale by growing the market share, as revenue = market share x unit profit. #strategy

On which side will you pick? Google who has been sitting on the chair of high profit margin (dominating the tech world by acquisition and direct competition) or #startups embracing ChatGPT like Swiss knife?

A question to all #investors, board members, CEOs, #entrepreneurs, #founders: how does #chatGPT impact the business model? How to embrace the challenge and turn it into opportunity, before competitors? (Disruption could come at any angle now)

why chatGPT works

Knowledge might just be the correlation between terms (embeddings)
Meme/concepts are the nodes in this graph (we may extract some level of knowledge graph from the model)
LLM/statistical model will be limited by the far-fetch in deepest domain
Modality and symbolic learning might be the next step to build better model?

The generalization ability of chatGPT-like LLM may come from the corpus data's quality, quantity, and diversity (aka, modality). And it will also be restricted by the collective knowledge such text-based data could represent.

If we could use a ring to represent basic human knowledge, the spikes on a such ring could represent the domain knowledge in each different area.

The max capacity of such "statistics"-based models would be extracting the underlying patterns discovered by gifted researchers from well-studied areas to those correlated but under-studied areas.

Nevertheless, it is a huge step forward. To me, it is breaking the cognitive imbalance (compared to ICT breaking the information asymmetry).

where chatGPT doesn't work

architectural

Any prompt that is not #flat. The attention mechanism + feed-forward only design determines the tokens will only be looked once (#YOLO).

Any higher-order computations like count the numbers between a and b will require some sort of recurrent operations (like for-loop).

Comparing it with programming language, the current chatGPT model is #flat in a sense that it can handle sequential, if-else logic, but not yet for-while loops.

My limited industrial exposure says that most of the current machine-learning models are statistical based. chatGPT as an example, the tokens are represented by embedding. The model is excellent in addressing the correlations.

But #correlation is not #causality. Flat correlation is the foundation of knowledge, and only as the foundation of knowledge.

The one different approach that works quite well for now is Wolfram, a symbolic approach of learning.

operational

Safety vs Iteration Cost/Benefit tradeoff

It's a single snapshot of the filtered internet for now, and for the decades to come with the current approach of #instructGPT and #RFHF.

Online LLM will be hard and expensive to prevent misuse and attack.

If OpenAI wants to use the data collected from public access for the next iteration of model training, it would take much effort to clean the data. It is natural that a more performing (valuable) model would be most sought-after for #cyberattacks (extremely high ROI).

The type and scale of attack could be beyond the movie "#Inception".

alignment

Blowing the whistle:

这大概跟仿生机器人瘆人有些异曲同工：那些不经意间的不自然，最是动摇心理。（我咋就信了这邪了呢？/非从业人员收到骚扰电话聊到后来发现是机器人的细思恐极症）希望关于 chatGPT 之类算法的 AI 伦理问题只是迟来而不是被忽略了… 希望这隐晦的吹哨不会被 AGI overlord 追究上 —— 别以为自己只是给了点训练数据，最后却发现被“洗了脑”，失去了信心。至少目前还学不会那些人的缺陷 😂

openAI issues

[business] alignment is a dynamic process, which will never be perfect to all eyes. Internal team alignment, as well as alignment with broader stakeholders is still an unsolved problem in current corporation management theory.
[business] alignment decision is a cognitive heavy task by nature. It is no surprise that only educated may follow and the elites are involved. Needless to say, there will be intrinsic selection bias. Not to mention to attention churn as the alignment constantly evolve and deepen.
[product] expensive (to clean the data for model iteration) and moderate (censorship with data cleaning strategy will regulate the model to be moderate)
[product] I don't see openAI has any agenda on decentralizing the alignment. Users are excluded in the discussion and the approach is not transparent enough for public auditing. There will also be trust issue as openAI transits into a "cap-profit" company.
[UX] loss of novelty due to slow model iterations (on top of 1, + hard to prevent issues, similar to the idea). Users are hard to be pleased continuously, their attention spans are limited with all those on-demand content applications.
[UX] the cost to store user session data across model iterations will be expensive. Such context will be forgotten by chatGPT, but not for one particular user. Thus, it won't be very personalized on top of the inherent flaw of statistical model.
[Sam Altman] despite all the effort, the pursuit of Universal Basic Income has not be really successful. At his position, not having a good solution for the social-economic segregation caused by inequality in education resources, and potential AI accessibility will wake him up at night. The #UBI is probably the most idealist solution his group could imagine, again due to the "survival bias" among the elites.

beyond chatGPT

多元化是被低估（或者是没办法被 tradeoff）的属性，无论 modality 还是 AI ethics 或者 equality

I could never value the diversity enough. It's the most important dimension, no matter in data modality, AI ethics or equality domain.

On top of diversity, symbolic learning and robotics will be interesting directions.

Here is my joke post:

My essay to build #AGI: build a humanoid with enough sensors & actuators (to learn perception), train a randomized LLM model with pre-trained GPT model (transfer learning on language corpus). Send to grad school/lab.
— Domi Jin (@domij_info) February 15, 2023

My master plan to build AGI is still waiting as that's not the most important thing I want to do now.

I'm happy to share that with a reputable party or for the resources I need to build what I mostly urge for now:

Why @sama? 1. Cap-profit is better than for-profit in #AGI space 2. #YC SUS alumni 3. Superhuman with respectful moral and ego 4. Low chance to have conflict of interests (my roadmap towards social good is different) 5. I can use some help with collaboration 2/2 (my DM is open)
— Domi Jin (@domij_info) February 19, 2023

how far are we from AGI?

If I would have to guess, the amount of energy it needs to train a human-level model would be close to 3.4E15 EJ. Total solar energy on earth is 3.4E6 EJ/yr. Earth age 4.5B yr. It may not need to start from scratch, as it can be built upon our intelligence.
— Domi Jin (@domij_info) February 15, 2023

The short-cut is to re-produce such evolution (model iteration) by transfer learning on human civilization. I will be a valuable accelerator in the process.

where do I stand now?

I saw it coming in 2018, however I didn’t ride along for visa and family reason. Well. Never too late to start again! https://t.co/bek1pbmkv4
— Domi Jin (@domij_info) February 18, 2023

See my brief story in the last few years.

where will I be heading to?

I'm still having the same roadmap as I refined in 2018. The only difference is that I will prioritize the 3rd one (education).

On a more tangible measure: I would like to achieve freedom (independence) of choices (financially, intellectually) by 2025.

My motto in Chinese has been updated since I married my wife. It is the Chinese saying of 但行好事莫问前程.

I would hope that my practice on "use technology to empowering people to collaborate better" can enable many to live a free (choice independent) life sustainably.

It is commonly said that financial independence -> free life. I see it more often as free life -> financial independence. Self motivation and actuation is the most powerful thing I could see in every living person. Unleash such potential is the key propaganda of my attempt in education and entrepreneurship.

Hello World Again

2023-02-21T00:00:00-06:00

I'm making a return to the site! chatGPT is just way too important to ignore for what I'm doing. See my recent post on chatGPT.

2018

I have a roadmap to build true #AGI.
— Domi Jin (@domij_info) November 28, 2018

This tweet (about start early on building #AGI with an extreme large model and high quality data, until becoming the darling of the arms race over the time) was posted on 2018 Nov 28.

Looking back, it is quite naive and shallow, as one cannot beat "hitting the ascending spiral of product-market fit", like a "runaway explosion" in stellar dynamics.

2019

At the time I graduated, the world in my eye was full of opportunities. I spent days and nights polishing the roadmap of life and decided to devote myself to empowering people to collaborate better. http://domij.info/

There are many paths toward the north star. I'm glad that I've never drifted too far from this journey.

I did my 101 learning on #entrepreneurship with #YC Startup School and founded a #startup called MindWave X as practice. I was clueless, when the immigration issue presses.

2019 - 2022

Fortunately, I met a great team (too many to endorse) and chose to join #Miles at the pre-seed stage. I threw all my passion into growing with the team. We set a foot in the real-time geo-locational advertising space. I called it as "challenging the consumerism & attention economy model epitomized by Google", by pursuing a "convenience" economy model, powered by high relevance on-demand advertising.

2022 - 2023

When family priority weights in on 2022, I am grateful to be reached out by Camille Parenteau, Erich Zancanella and Roma Badiani from #Thumbtack. I couldn't ask more support to paddle through this period (full gratitude to my manager Bethan Nichols, Richard Demsyn-Jones and Denys Kopiychenko), while experiencing the future of work, pioneering in remote collaboration, at a decentralized but autonomous late-stage company.

I'm thrilled to work with some of the most kind and talented people, on learning and regulating the local service market, to reach the full efficiency, by aligning the interest and empowering different parties, to make informed decisions.

Now

This is just the start of the journey of #collaboration and #empowering. I cannot be more excited now!

where will I be heading to? (cross posting)

I'm still having the same roadmap as I refined in 2018. The only difference is that I will prioritize the 3rd one (education).

On a more tangible measure: I would like to achieve freedom (independence) of choices (financially, intellectually) by 2025.

My motto in Chinese has been updated since I married my wife. It is the Chinese saying of 但行好事莫问前程.

I would hope that my practice on "use technology to empowering people to collaborate better" can enable many to live a free (choice independent) life sustainably.

Free Market in a Possibility Universe

2019-09-27T00:00:00-05:00

Declare: While paving my way towards my neverland, I don't mind being alone. I won't return to advertise and persuade either. Because the trace I left is not meant to guide, it is to inspire.

Again, this starts with a stream of morning thoughts posted on Twitter, after reading a post about the potential of LPR (loan premier rate).

Not sure if I understand it correctly. LPR makes China sounds like national capitalism. It is definitely not the reflection of the free market. But this might work out the best when the information barrier and decision skill of the state is far advanced than the people, and most of the corporations.

Of cause, it could be the consequence, and source of rent-seeking, which officially begins as let some be rich first, and let the early rich help the rest.

I don’t think any gov can do better than PRC. Such policies can only work in this rare case, where one party can hold its principles consistent most of the time.

It is inspiring for anyone to wonder how the first awake ones should treat the rest. Since we fancy harmony, I hope that we will, when the time comes that the rest are also awakened, the ego of the elites will not feel hurt not to take advantage of the so-then-called historical market power and become the blocker.

That moment could be the rise of an AI overload, which efficiently negotiating the supply/demand to maximize resource distribution.

But computationally, an agency at that capability is much more difficult to build than multiple at a lower level. When multiple AI overloads exist, will that world fancy equal right? Will they adopt the free market of power supply, network bandwidth, and computation resource? Unless there is shared memory or more that I don't have the imagination about, it will be just another Darwinism world.

The world for that type of intelligence will be no longer limited to the earth ecosystem. Scale-out or scale-up are both not radical solutions. It is only a delay of the zero-sum/limited resource conflict.

The current issue is that, as the entropy increases regardless of any ideologies, many awaken-but-not-wise-enough are losing faith. We haven’t tried enough and there are more means than possible to be explored.

In this preference-less world that is full of possibilities, what is the loss of annihilation, when one surely know, support, and appreciate the existence of others? The odds of possibilities are never in favor of one, and it is the choice to be a different one and embrace the many that matters. We are no better compared to the wise ancestors in this kind of spiritual realism.

The arrogance built upon science/technology development is the biggest obstruct. Straits is the gate. We need another Renaissance to fairly position the logical reasoning behind, back to the spotlight of civilization advancement.

Advance inductive and deductive reasoning, while collaborating on building a shared knowledge base until more enlightened. That is what I forged into my life mission, by the means of crafting a collaborative society, MindWave, and so on.

Exhausted but sprinkling with the joy of better storytelling skills, to express, to be heard and to be enlightened. All words are mine, biased by the possibility I experienced and expected.

Now I recon the younger me set the goal as to see the world of all possibility is what sets the tone. X beyond the mountains, Y through the seas, Z to the stars.

Thoughts on Autonumous driving and Humanity

2019-09-25T00:00:00-05:00

When autonomous driving cars start to reach the critical mass, AV algorithms will start to turn from defensive into aggressive, slowly outpacing human drivers, and then rapidly racing over AV algorithms themselves.

Because defensive algorithms are conservative. One can develop a simple AV algorithm and it will be safe when other cars with advanced algorithms will avoid putting each other in danger. It is a matter of time for AVs to become more aggressive to maximize the benefits of road priority.

In a world with only edge computing, if there is no self-regulation, the road will become the new battlefield of a dystopian world, for AV companies.

Essentially, it is a debate about human governance or machine governance. The latter will never win unless we trust machines even when we don’t understand why they make the decision, which is only one lawsuit away.

We thought it will be machine-assisted human governance. But it won’t take long for machines to take over, as the nature of the free market is a Darwinian world evolving towards resource efficiency (entropy). It is more likely to be human-assisted machine governance until we are doomed.

If AV can walk away from court without human reasoning-able causes in any case. It is a matter of belief that we’ve given away our right to justification. Not necessary a bad ending, but we should be cautious.

Because computation never has any moral goals and the god of possibility has no preference. The only reason machine governance becomes human-friendly is by crafting the human principles and human values into it.

By doing that, we are basically digitalizing ourselves or manufacturing cyber humans. But because it is so hard for us to collaborate within ourselves now, I don't believe it will be easier at the time.

It takes more than one prophet to make the world awake, and it takes more to raise and make changes matter. Are you, willing to be Prometheus?

Fix ourselves is the only way to fix the world.

Manifesto

2019-03-29T00:00:00-05:00

The Manifesto

The Vision: Technically Empower People to Collaborate Better

Principle to Collaborate

Autonomy
- Social Contracts
- Code of Conduct/Principles
  - We are a family, community, crowd-owners of a social company
  - Be bold, radical solution!
  - Be modest, I know that I know nothing.
  - Be honest and embrace changes, open source and open management.
Open Source, no NCA, NDA
- Open Meeting
- Open Decision
- Open Development
Open Management
- Peer Review
- Jury Decision
- Liquid Democracy

One can make the shape, but it is all that can fill the gap.

Collaborative Society

The largest achievement and challenge for humans is how to collaborate better. It could be broken down into two tasks, 1. how to communicate better (effective & efficient); 2. how to organize better (optimized team & motivating).

When it comes down to product-market fit, there is a lot of engineering work to be done and an even greater challenge to recruit and manage a team to execute. Technologies are only byproducts and tools. It is how everyone is valued and matched to take ownership to develop that really strikes. That’s what put my two pieces of challenges together to help people collaborate better.

When we talk about social identity and job market, resumes and social media always come to our minds. But these are projections that don’t fully nor correctly reveal our personalities, knowledge bases, or abilities. How could that result in the most optimized team formation and function? It becomes natural to develop solutions like MindWave to help people communicate, and help understand communication while eventually understanding people as well. Of course, it depends a lot on the technologies like signal processing, ASR, NLP, knowledge graph, cloud services as well as adoptions for different domains and scenarios. But it is how people communicate their thoughts and how people organize to deliver the biggest assets of human civilization. With the service and tools to understand how people communicate, it becomes practical to effectively evaluate the contribution and distribute the ownership accordingly. Mild management or no management will ever be needed if collective decisions become an obligation as well as a reward corresponding to the dynamic credibility and ownership, which could be realized with a blockchain design of liquid democracy and decentralized social credit system.

As until now, I feel somewhat comfortable and satisfied to fully push forward with this vision to empower people to collaborate better. The first step of collaboration is always to communicate (openly) and embrace changes.

If this somehow resonates with you, please step up to learn about how to get invovled.

协作社会

我现在认识到我真正要达成的是促进人类更好的协作。其中的挑战包括 - 促进有效的沟通， - 优化和激励管理。

无论是通过硬件产品帮助会议记录，还是SaaS服务促进办公自动化，又或者是通过写关于未来的科幻故事。

当然，在切入市场的时候，会有很多的工程任务。更大的困难还在于招募和管理一个团队。这种时候，技术只是工具和资产。真正重要的是让团队各尽其能，默契一致。这也是我为何将促进人类更好的协作，细化为两个方面。

当我们讨论社会身份和就业市场的时候，通常考虑的是简历和互联网社交。然而这些只是我们个性，知识以及能力的一个缩影，甚至都算不上准确。这种情况下，组建和实现最佳的团队就成了困难。MindWave自然变成为了最自然的解决的方案。通过帮助人们更好的沟通，并理解沟通，也能最终对沟通的人有更好的画像。当然了，这个项目需要依赖各种技术，比如信号处理，语音识别，语义理解，知识图谱，云服务以及不同领域和场景的定制化。但正是交流和组织背后展现的思维和合作才是人类文明最大的财富。有了促进人类沟通的服务和工具，通过量化评估各人的贡献实现匹配的分配才变得可能。当集体决策与个人的义务和分配所得相对应的时候，人们自然地会以其社会信用和责任理性的参与集体决策。这样的管理机制可以通过基于区块链的去中心化社会信用体系以及流动民主来实现。

意识到了这些，我现在才觉得有足够的思考底气去全面推进这个促成人类更好协作的理念。而协作的第一步永远是开诚布公地去交流和拥抱异见。所以我很期待你的反馈。我目前已经积累了一些人脉但是依然需要更多的帮助去实现之后的步骤。

如果你恰好也有这样的理想，我非常期待能与你一起成长和实现。

Response to 996

你们需要的是更多“站着挣钱”的公司。

公司是大家剩余价值的沉淀，分配制度没有变革的情况下，是无法调和阶级矛盾的。持股无非只是努力从被剥削者成为剥削者而已。要radical的解决这种矛盾，就需要有新的协作机制。

人类有天然的生理限制，包括 1. 有限的主意力（信息IO） 2. 有限的思维能力（信息processing）。先不提衰老，这些限制是以生物进化作为时标。而各领域技术创新，是与信息分享（联通率）和网络规模（人口）成正比，是社会发展作为时标。这两者的时标差异必然导致互联网信息爆炸，乃至技术焦虑。要避免这种焦虑，又得回到分配模式的变革以及用新技术接触对技能的依赖。AI的时代，我们需要的是一场文艺复新，回归人本主义。摆脱技能和分配对劳动力的束缚，进一步解放创新能力和协作规模去实现进一步的生产发展。

我正在力图解决这一挑战，通过技术实现 1. 有效全面的沟通 2. 优化和激励管理。

Competitive Advantage vs Absolute Advantage

2019-01-10T00:00:00-06:00

It is hard to fully evaluate a person. The most conventional way is the lookup the education history and career experience from resume. It provides a quick reference of the ability based on the established institutional reputation. This efficient but coarse filter to position a new candidate, needs additional validation to be followed to fine grain one’s knowledge abundance and skill level. This is the epistemology in recruitment field. The fitness is a measurement of competitive advantage at that moment. The dynamics of potential realization and future achievement is still mystery and impractical to incorporate.

In social networking, connections become the endorsement of one’s personality, sociological status, economical ability and political attitude. While we cherish the trustability built upon such underwriting connections, we take the detail as privacy. The psychological comfort cannot be fulfilled with unconditional transparency. The only motivation to withdraw one’s conservation on that is to initiate a new connection, or “relation of production”. In a perfect world, not only is everyone born free and equal, but also has the same subjectivity and initiative. We cannot deny the efficiency promoted by scale effect and deep domain expertise, where division of labour naturally differentiates the population, along with the geographically and politically affected globalization. The relation of production shapes the distribution of production and beyond with free market.

As the understanding and toolsets advance, things may change and should be. With machinery, we overcome the productivity bias induced by biological difference from age, sex or physiological condition. With information and communication technology, we reduce the informative decision making bias, regardless of the geographic difference on cultural awareness, development status, or knowledge access. But the further development in capitalism encourages the competitive advantage rather than the “absolute advantage” of a personal, which values more on bolder knowledge span, creativity and enthusiasm. In the competitive world, rule of limited-sum game optimizes an individual to fit into a discriminative society. Privacy inequality is thus a reflection of the sociological bias.

At one glance, it seems that continue the development to increase the total sum is the silver bullet. I’d only point out the nature limit of material and energy resources won’t be easily lifted at foreseeable future. Space technology and fundamental physics will eventually be the bottleneck. But at the moment, it is the mode of distribution. There is enough food to end hunger and there is more plastic than needed. Similar to the information bias in imperfect market, the evaluation bias to monetize the ability of a person causes the inequality. It is far too early to claim any solution. But machine learning and artificial intelligence is reducing the disorder of market with informative decisions followed by rational quantitative analysis under econometrics. Heterogeneous supply and perfect information may not be a dream in the future of decentralized democracy.

White paper - A future imaged

2018-11-27T00:00:00-06:00

Table of contents:

Manufacturing
Socio-economy
Techinical Roadmap

Manufacturing

Uncapped productivity for material abundance. Socks for example
On-demand: when land in airport, pick up new socks for duration of the trip, drop off socks when return
Material efficiency: socks are made with dissolving fibers, produce on-demand, recycle as needed (like 3D printing & material recycle), less shipping/packaging cost
Energy efficiency: logistic/inventory cost is building up, compared to the scale effect from division of labor & mass production efficiency, under global distribution; made in China is getting more expensive, and so it will be for made in any other emerging market; Amazon will be threaten when new retail (self-service store & self-delivery) + local supply made cheaper with on-demand manufacturing. I.E. socks printed at airport, usb chargers re-engineer/recycle/share, old clothes redesign/recycle
Customization: design, style, taste is the reflection of personality. For socks under the new manufacturing, consumers shop design/style in e-marketplace, feed their biometrics to let AI adjust to fit, modify color/material, and send to local hub for print/assemble/“bake” using 3D printing, robotic arms, biological engineering or any other techniques.
Ownership rethink
Benefit of privacy: we can’t share socks or undies for hygiene reasons, but more for psychological reasons, which can be solved with proper sterilization and psychological “placebos”.
Benefit of convenience: my primary reason for having a car is the trunk space, to store things that feel needed but not often used. It may be replaced with modularized self-walking luggage.
Affordability: we rent, when the cost of ownership is too high for the usage. But ownership comes with responsibility. We provide space, care and proper treatment for pets. Stuffs we own, will also occupy space, need regular care and proper disposal. Otherwise, the space, material and energy are wasted.
Summarize
Current manufacturing is not energy/material friendly. A lot of resources are dissipated/wasted in logistics: inventory, shipping, packaging.
We can’t find/afford things exactly as we want in the current manufacturing model.
Efficiency of logistics come with huge delivery cost.
Hierarchical manufacturing hub that feed on basic materials, produce on-demand, recycle as needed, with marketplace for designs can maximize the energy utilization and ensure material abundance.

Socio-economy

Productivity supplies distribution.
[old age] Autarky, self-sufficient production with limited exchange and community market.
[past] Electricity lifts off labor-bias with machinery, labor cost per production drops and productivity blooms.
[now] ICT lifts off information-bias with internet, market grows globally and circulation efficiency leaps with e-commerce,
[future] AGI will lift off skill-bias, domain knowledge and expertise will be less dependent to produce better products and services. Any nurse with medical AI can perform high-quality diagnosis and treatment plans.
Distribution shapes the socio-economy.
[old age] Reputation. Culture and traditions are conventional agreements to regulate individual with the tribe, community or society
[past] Power. Darwin society with expiration, monopoly and crime leads to slavery, militarism, imperialism, Fascist.
[now] Network. As peaces continue, social classes consolidates, so it is with wealth distribution. Capitalism leaves less chances for the resourceless families. It compliments the elites in Wall Street and Silicon Valley but many feel anxious about the fast-changing job market and heating threats from AI powered automations. That’s how I interpret the political shifts due to the rising voice from the worried/underappreciated workers.
[future] Human first, AI is mimic of human intelligence and the unprecedented computational ability should aim to enhance human. Let AGI be good at information processing and empower human to take responsibility of moral consideration, decision making, medical result delivery, providing humanize service and creativity. Follow this direction, individuals will no longer be bounded with single career/job function, to accumulate competitive advantage for financial reward. Income differences will also decrease, as personal domain knowledge and expertise cannot compete with AGI. Equality will detour individual’s interest to creativity, personal development and social good.
City and community
Declare
- The diverse possibility is the beauty of future.
- One man’s expectation can’t and won’t satisfy all. But such imaged future is what my technical roadmap can leads to.
Community Management
- Every social transaction (behavior) satisfies the consented social contract adds personal credibility on his social credit.
- Example: when the community agrees on sterilizing all house cats. C tells D that he feels lazy to sterilize his cat and it won’t matter. AGI notices the violation of consented social contract and lower C’s social credit correspondingly to reflect his credibility.
- The math behind: enforcement is hard/costly, and legislation is dangerous with central authority.
- The average social credit profile of the members represents the credibility of the community, which goes up to higher level of communities like city, state, continent, planet and more.
- Unlike black mirror - the future of social media, the enforcement is AGI automated to remove human bias.
- Every proposed change of social contract in the community will be voted and adopted automatically, using voice based human-information-interface.
- Like-minded people gather to form community, guided by democratically voted social contracts, which are registered as smart contracts on community blockchains.
- Example: A tells B that they should sterilize house cats and B agrees to advocate the policy to community. Wearable device with AGI automatically recognize and draft a proposal to the community. A&B discusses about the detail and let AGI make a campaign. Members who see the campaign may express their options, which will be collected and counted. Soon, AGI receives enough feedback and make the agreement as official social contract.
- The math behind: AGI can automate and decentralize legislation processes with transparency and efficiency; it is easier to understand legislation in physical world but easier to adopt in virtual world with smart contract on blockchain; one approach is to establish a decentralized identity system to evaluate individual’s social credit from TOC histories. Anyhow, a society with trust is a species will last.
Summarize
Hierarchical manufacturing hub for material supply
Open government powered by AGI to ensure enforcement efficiency and transparency
- Social contract on blockchain for legislation
- OMO credit system for personal credibility
AGI to empower people the skill/experience needed for careers, people can focus on personal fulfillment and sociological development
With lower qualification requirement, equality will be easier to achieve during distribution, better humanity will be encouraged in society build with trust

Techinical Roadmap

Initiatives on Sociological Practice

2018-11-19T00:00:00-06:00

This is my open letter to Stephen Wolfram, who is a living inspiration. But I'm glad to share with any like-minded person, who doesn't "live the moment", who longs for the future, who cares about humanity, and so on.

I’m always anxious, about people’s lack of first principle thinking, the reckless to the environmental and sociological consequences.

Humanity and Human Intelligence are beautiful myth, that lifts homo species off the biological evolution track, into the fast track of sociological evolution. We advance so fast in science, technology and social organization in the scope of earth history, which drains the nature resources exponentially. From animal power, to chemical power, to nuclear power, the ability of energy utilization marks our level of civilization. The increase of productivity, also shifts the political landscape by socio-economy effects.

In the new era of ICT and AGI, a new scope, computation consumption per person, may rise as a new indicator to better segment our development stage. Despite the excitement, in front we face global warming and the subsequent natural disaster, unintended simplification of ecosystem and subsequent fatal plague due to globalization, foreseeable energy crisis & waste cycling & astronomical events, not even to mention the growing inequality in wealth distribution & jobless & fear of jobless & evil AI...

Selfishly, I don’t want to live in such a future, nor for my future generations. Nuclear fusion, AGI can power the productivity and economic growth further more, but not if the humanity fall due to the sociological consequences. That’s what puzzles me over the years. I have decided to develop a few approaches in hope to solve that. I haven’t lived a comfortable moment yet, as a second born from a farmer family in China, but I won’t end with a decent job for comfort.

I sincerely would like your advice, on how should I validate my approaches, find resources to develop and make efforts to solve, or inspire more to work on a solution. I’d love to discuss and share my approaches in detail, which may not be fit in one plain email for all the hypothesis, experiment design, business plan, R&D roadmap, technical dependencies, potential obstacles and sociological consequences I’ve pictured.

You may find more about me at domij.info, and really look forward to meeting you in future.

Decentralize social credit + identification system

2018-10-28T00:00:00-05:00

This idea has been in my mind for a while. I'm worried about the potential consequence that may go north if carried out in the wrong hand. But similar things are happening already and life always find its way. We won't make progress without bearing risks. And I hold faith in collective humanity.

The project

Modularized Terms and Conditions that makes legal consents clear and easy, just like grant permission for mobile apps.

What's new

Our attention is limited and precious
NLP can break down page-long legal bindings
Consent to one agreement at need is easy
Universal consents can be done OAFA
Valued attention will help iterate agreements
WE build on social contracts and universal value

MVP approach

crawl Terms & Conditions, Privacy Policies, User Agreements over the Internet
clean and unify data format
train NLP model on collected data plus legal documents ad-hoc
topic model each sentence/paragraph for all the files
term/topic weight based on similarity/quote or page-ranking methods
generate first version of modularized terms and conditions

Use case example

free dev API for website/software/service developers to quote and assemble their initial legal documents.
free user API for consents management and authorization management.
paid underwriting service for any potential cost in litigation.
smart contract of user-consent-provider on blockchain to make it decentralized.
decentralization will make terms and conditions more standardized.
unique identification based on time-stamped consent history built upon blockchain.
digitize paper-based legal system.
regulate virtual behavior of any entity with the social impact of an universal identification system.
legislation made easy with true votes backed by decentralized social credit system, from personal social credit, to community social credit, regional social credit and more.
rebuild democracy with the spirit of social contract based on each citizen.

If you have the same interest, or would like to take it off with some sustainable business plan only for practical reason, I'm here to share my 2 cents. At the moment, I will be taking responsibility of my share holders on a career to enhance humanity with technical assistance.

Some Pitch words

The constitution of unites states is one of the most inspiring achievements in human civilization, especially at this decisive moment of history.

I’m bringing an approach towards decentralized democracy to help re-craft social contracts (pointed out by Jean Rousseau) that shape the modern society.

I want to launch a NGO project for decentralized democracy.

Start by building a system that provides standard/modular terms and conditions for suppliers/ dev teams to quote and manage users’ consents. So that user only need to read and consent new additional terms at need and dev teams can save effort on legal concern. Just like permission to photos is only required when you want to send a picture in the app.

On the other side, the unique time-stamped user consents in our system can become an identity that is hard to compromise. It will be extended into physical world for any agreement, marriage vow, or security check to transform ethical bonding into a measurable social contract. In which, will evolve and expand to be community rules, cultural traditions, national constitution and universal value.

I will demonstrate how to build this system with natural language processing plus blockchain techniques, with features like “learning” curve and Page ranking to let the community to balance and involve in the legislation of a decentralized social credit system, which I hope will be a Renaissance of the United States constitution.

PS

You may support by sharing your thoughts with twitter trend #changeTOC

Lessons that every startup should learn early

2018-10-18T00:00:00-05:00

These are my notes learned from reading. I would have done many things differently. There is no rush to broadcast one's eager and anxiety at start. The real service and product that may help make a difference is what one should really aim on.

I will stop twittering or posting on LinkedIn as there is nothing needs to prove yet. The work will prove the value itself and I can do it, just with a few really good friends who share the same ambition. That's enough to start. Anything add-on will be extra, but it's not the main cause to chase for.

Startup school library

The Pocket Guide of Essential YC Advice, Michael

Launch now
Build something people want
Do things that don’t scale
Find the 90 / 10 solution
Find 10-100 customers who love your product
All startups are badly broken at some point
Write code – talk to users
“It’s not your money”
Growth is the result of a great product not the precursor
Don’t scale your team/product until you have built something people want
Valuation is not equal to success or even probability of success
Avoid long negotiated deals with big customers if you can
Avoid big company corporate development queries – they will only waste time
Avoid conferences unless they are the best way to get customers
Pre-product market fit – do things that don’t scale: remain small/nimble
Startups can only solve one problem well at any given time
Founder relationships matter more than you think
Sometimes you need to fire your customers (they might be killing you)
Ignore your competitors, you will more likely die of suicide than murder
Most companies don’t die because they run out of money
Be nice! Or at least don’t be a jerk
Get sleep and exercise – take care of yourself

Do Things That Don't Scale, Paul - recruit users manually - weekly growth rate - insanely attention to users - pull a Meraki: manually manufacture - launch doesn't matter - Partnerships too usually don't work

In the best case, both components of the vector contribute to your company's DNA: the unscalable things you have to do to get started are not merely a necessary evil, but change the company permanently for the better. If you have to be aggressive about user acquisition when you're small, you'll probably still be aggressive when you're big. If you have to manufacture your own hardware, or use your software on users' behalf, you'll learn things you couldn't have learned otherwise. And most importantly, if you have to work hard to delight users when you only have a handful of them, you'll keep doing it when you have a lot.

Jessica Livingston’s Pretty Complete List on How Not to Fail , Jessica

Make something people want.
Stay focused.
Don’t worry about being a woman. in our case: don't underestimate woman
Measure your growth.
Know if you’re default alive.
Keep expenses low.
Fund-raising gets harder. Series A investors are looking for performance.

If you start by making something people actually want, focus on making users happy, make sure you have a good growth rate and don’t over-hire, you’ll be in a very happy position. You will be master of your own fate in a way that very few people ever get to be.

Do not be distracted on these - “Grabbing coffee” with investors - Talking with potential acquires - Networking - Recruiting boards of advisors - Doing a “partnership,” thinking it will get you more users - Spending time on PR before you have made something people want - Arguing on social media - Going to conferences - Worrying about being a woman in tech

Users You don't want, Michael

small group may deviate your focus
evaluate the need but don't ignore it
prioritize and say no

Startup Priorities, Geoff

ask how should I be spending my time: product and customer satisfaction in my case
Choose a key metric to track and focus exclusively on making that metric grow: average use time
choose the activity that you believe will directly result in increased growth of your chosen metric

A simple metric: $(b * d) / c$

$b$ tells you how many of your users, current or new, will be impacted by a new feature, $d$ describes how important that feature will be to the average user, and $c$ gives you how hard it is to build the feature. Just like MVP for each phase.

how to raise a seed round, Geoff

bootstrap or seed round

why

cash for grow
war chest
- hiring key staff
- public relations
- marketing
- sales
brutal: long, arduous, complex, ego deflating

when: usually when you can, you should

story and reputation
idea, product, some amount of customer adoptions
product market fit
growth rate of 10% per week for several weeks

how much to raise

best: raise once and for all, avoid more than 25%
second: follow-on round
create multiple plans assuming different amounts
set an essential amount to succeed, aim for faster grow
how many months of operation
- $15k per month for an engineer
- 18 month
- you are raising for N months (usually 12-18) and will thus need $X, where X will usually be between $500k and $1.5 million.

Financing options: First comes a seed round, then a Series A, then a Series B, then a Series C, and so on to acquisition or IPO

Most seed rounds, at least in Silicon Valley, are now structured as either convertible debt or simple agreements for future equity (safes)

Convertible Debt: loan with interest rate + maturity date
- it converts to equity (thus, “convertible”) when the company does an equity financing
- usually have a “Cap” or “Target Valuation” and / or a discount
- A Cap is the maximum effective valuation that the owner of the note will pay, regardless of the valuation of the round in which the note converts
- a discount defines a lower effective valuation via a percentage off the round valuation
- Convertible debt may be called at maturity, at which time it must be repaid with earned interest, although investors are often willing to extend the maturity dates on notes.
Safe
- A safe acts like convertible debt without the interest rate, maturity, and repayment requirement.
- safe primer by YC
Equity
- setting a valuation for your company
- hire a lawyer when planning to issue equity
- dilution based on new equity
- equity incentive plans (option pools), liquidation preferences, anti-dilution rights, protective provisions, and more.
- always best to use well-known financing documents like YC’s safe. These documents are well understood by the investor community, and have been drafted to be fair, yet founder friendly.
Valuation
- let the market set your price and to find an investor to set the price or cap
- The more investor interest your company generates, the higher your value will trend.
- not to over-optimize
- find a valuation with which you are comfortable
Investors: Angels & VC
- angels are amateurs and VCs are pros
- faster and more mature

Documents needed

Your company / Logo / Tag Line
Your Vision – Your most expansive take on why your new company exists.
The Problem – What are you solving for the customer–where is their pain?
The Customer – Who are they and perhaps how will you reach them?
The Solution – What you have created and why now is the right time.
The (huge) Market you are addressing – Total Available Market (TAM) >$1B if possible. Include the most persuasive evidence you have that this is real.
Market Landscape – including competition, macro trends, etc. Is there any insight you have that others do not?
Current Traction – list key stats / plans for scaling and future customer acquisition.
Business model – how users translate to revenue. Actuals, plans, hopes.
Team – who you are, where you come from and why you have what it takes to succeed. Pics and bios okay. Specify roles.
Summary – 3-5 key takeaways (market size, key product insight, traction)
Fundraising – Include what you have already raised and what you are planning to raise now. Any financial projections may go here as well. You can optionally include a summary product roadmap (6 quarters max) indicating what an investment buys.

Fundraising Rules to Follow

Get fundraising over as soon as possible, and get back to building your product and company, but also…
Don’t stop raising money too soon. If fundraising is difficult, keep fighting and stay alive.
When raising, be “greedy”: breadth-first search weighted by expected value 2. This means talk to as many people as you can, prioritizing the ones that are likely to close.
Once someone says yes, don’t delay. Get docs signed and the money in the bank as soon as possible.
Always hustle for leads. If you are the hottest deal of the hour, that’s great, but everyone else needs to work like crazy to get angels and other venture investors interested.
Never screw anyone over. Hold yourself and others on your team to the highest ethical standards. The Valley is a very small place, and a bad reputation is difficult to repair. Play it straight and you will never regret it. You’ll feel better for it, too.
Investors have a lot of different ways to say no. The hardest thing for an entrepreneur is understanding when they are being turned down and being okay with it. PG likes to say, “If the soda is empty, stop making that awful sucking sound with the straw.” But remember that they might be a “yes” another time, so part on the best possible terms.
Develop a style that fits you and your company.
Stay organized. Co-founders should split tasks where possible. If necessary, use software like Asana to keep track of deals.
Have a thick skin but strike the right balance between confidence and humility. And never be arrogant.

DON’T:

Be dishonest in any way
Be arrogant or unfriendly
Be overly aggressive
Seem indecisive – although it is okay to say you don’t know yet.
Talk so much they cannot get a word in edgewise
Be slow to follow-up or close a deal
Break an agreement, verbal or written
Create detailed financials
Use ridiculous / silly market size numbers without clear justification
Claim you know something that you don’t or be afraid to say you don’t know
Spend time on the obvious
Get caught up in unimportant minutiae – don’t let the meeting get away from you
Ask for an NDA
Try to play investors off each other when you are not a fundraising ninja
Try to negotiate in real-time
Over-optimize your valuation or worry too much about dilution
Take a “No” personally

mean people fail, Paul

Successful founders are usually good people

paths in tech industry, Michael

founder, executive, and employee

Strategy for People Who Want to be Founders Your initial goal is to accumulate the prerequisites to being a founder.

Identify potential teammates you can work with who have the required technical skills to help you build your MVP.
Figure out the financial plan. I.e. Do you have enough money in savings, do you have friends/family who can provide seed investment, can you bootstrap, can you reduce your spending and save enough give yourself 6-12 months to work on your idea?
Identify a problem that you and your potential teammates are passionate about solving.
The least important (but still required) is having an idea on how of how to solve the problem.

startup playbook Sam

Lecture 18: Legal and Accounting Basics for Startups, Kirsty, Carolynn

sign the purchase agreement
83 (b)
vesting, for the long haul
seed round: conversion notes, safes, no voting rights, cap
board seats: choose wisely, must add value
adviser: investor is de facto adviser
Pro-rata rights: rights to maintain percentage ownership
Information right: monthly update
company expenses
- separate and spend wisely
- bookkeeping the documents for tax
founder employment
- set-up payroll service
- worker's compensation insurance
- pay salaries
- minimum wage: $2000
hiring employees
- working right in US
- employee: responsible for taxes
- contractor: consulting contract, 1099, not responsible for taxes
- IP assignment
fire employees
- fire quickly and professionally
- communicate clearly: we're letting you go
- 3rd party
- pay the wage
- repurchase unvested stocks
legitimacy

CPA: tax return Lawyers: use standard forms first

technical co-founder Adora

Make a firm offer. If you don’t have money for salary and/or that person has no outside commitments requiring salary, then propose 50/50 equity split.

Lastly, when you identify the right person. Instead of pitching them directly your idea. Brainstorm ideas with that person so it evolves with the both of you. This will be a good start to a hopefully lasting relationship of building together.

More thoughts on Future Transporation

2018-10-17T00:00:00-05:00

Sometimes I just feel that self-driving car is too in-box thinking. @boringcompany might be more practical for its top-down design that reduces computations into realistic scale. CV & Lidar are comp-expensive, not to mention the scheduling that scale with N^2.

If we add the complexity of delay control, the system will be chaotic like the failure of jurassic park. Plus, any malfunction in the system will probably be a costly issue that no society can afford.

If it's only used in out-of-city transportation, then bullet train and @hyperloop will be more appealing. Of course, I enjoy the flexible #trunk space. What if we can modularize that and make it easy to plug?

What I really want to sell here is that, road is only 2 dimensional space that has limited capacity. We should utilize the airspace! I'm not talking about #FlyingCars by @TerrafugiaInc or #JetPack. They will fall and fail. And that will scare the world like #Concorde.

And the computation scale will be no less than self-driving cars. #KardashevScale is a measure of civilization's level of technological advancement based on energy usage. We should add sub scales based on the computation usage per person. @PeterDiamandis

Okay. Back to airspace usage, we can think in these directions, 1. separate power unit and passenger space, 2. use guided rails for power unit (top-down self-drive system), 3. let passenger capsule fly like a glider towed by the power unit. @elonmusk #FutureTransportation

The benefits are, 1. safer, glider can crash land and there is also a string to hang on; 2. rail for power unit is cheap to build and maintain (the weight load will be offset by lifting); 3. modularize capsules are simple and cheap with no complex control system needed; 4. we won't need to get down to ground anymore, only way towards billion-population city built vertically; 5. easy to adopt in #SpaceSettlement, you may need this on the #MarsColony, @elonmusk; 6. inspirational and worthy to pass on to the future generation.

What else? The scale will be like 10 FLOPD (flops per day) avg before calculator invented, 10^2 FLOPD before transistors invented, then it goes viral based on Moore's law.

From radio/DVD, to gaming console/computers, until smartphone with cloud service, our consumption of computational power is increasing exponentially, even faster than Moore's law with so many #SaaS and even #HaaS, hardware as a service like what we do in MindWave.

Birth of MindWave X

2018-10-13T00:00:00-05:00

To many young entrepreneurs, Y Combinator is the dream place to launch startups. I didn't know anything, not the startup schools, not the Hacker News, not the Office Hours, not the YC partners. I signed up the YC Hack 2018 just like a Texan town boy, without too many thoughts.

Things start to heat up when I joined the Slack, where like-minds start to pitch their ideas and put up their profiles. So I did the same thing and hoping to get some connections with the silicon valley people. Nothing much happened.

10/04/2018, it was a normal day in Texas, hot and sunny, when I drove from Brownsville to Austin for a flight to San Jose. The 24 hour journey is a mind-blower. I met with these two Denmark cowboys, who hacked a way to store data on blockchain with 5% of the normal gas fee. There are so many flaws in many of the picked projects but all those teams did very impressive delivery of some products or services. There is no cash award except for an interview with YC and some office hour, which I realized later on that it is exactly what a successful company needs, a good team with execution skills.

I didn't learn it that night and my ego is really somewhat annoying to others. But I also didn't stick to my plan and make it a go, as I feel tired of being the idea source plus the doer. There is no easy path as hitchhiking and everything I want have to be earned by own effort.

It is never late to take the lesson and here I am, as the founder of MindWave after one week. The initial plot is interestingly simple and attractive so that many have approached me to carry it on.

There is no other moment that cheers me up so much to work continuously with full passion and inspiration. Every step makes it closer to real and there is just no foreseeable limit of this journey that will drain my creativity.

Here I present, Mind Wave, a company that delivers the best of assistance to human intelligence.

We believe that Human Intelligence should come first. As human is still more intelligent and AI should aim to enhance that. AI is the new electricity but we will be the Renaissance of the revolution.

Request for Partnership

At MindWave, we seek to build an approach to understand the most intuitive and direct reflection of human intelligence, speech and conversations.

By providing Hardware as a Service, we will act as the upstream of human-internet interface that not only enhance your communication skills but also unprecedented add-on values to access, interact and understand informations.

From augmented intelligence, to artificial intelligence, MindWave will transform the world to rethink about data, device and intelligence.

We are a Y Combinator project, launching our first prototype by the end of October, with mapped out path towards marketing, Research and development with continuous revenue generation plan.

We are looking for Partnerships that can share our vision and help expedite our mission towards the Tomorrowland of Human Intelligence.

Thoughts on Artificial Intelligence and Natural Language Processing

2018-09-19T00:00:00-05:00

My experience of AI/NLP research and development is limited compared to HPC and data analysis. During my summer intern as R&D in NLP/query understanding, I studied linguistics and implemented the stoke and order into Chinese NLP, based on its nature as a logographic system. Time will reveal that even though computer programming is highly based on English, AI/NLP may work better in a different language system, which will have the elegant simplicity of mathematics and abundant expressiveness in Chinese.

Back to practice, I have been studying word embeddings like word2vec, and then utilize them for specific linguistic tasks like NER/POS with supervised LSTM + CRF model, or unsupervised models for topic modeling, text summary, etc. From my bilingual background, I realize that knowledge base could be further validated or expanded by multi-languages. I am also interested in the similarity between state-space model and neural network model. Despite the magic non-linearity from activation function, both are seeking optimal/equilibrium with updating techniques based on gradient (and chain rule). The advance of AI/NLP is thus mainly the result of Higher Performance Computing. Leapfrog, MCMC, gradient descent, SGD, batch updating or so are still some compromise regarding to high dimensional parameter space iteration, given the ever-increasing computation power.

Nevertheless, data quality is very important at the moment and it's the cornerstone for commercial products, depending on/deciding the use case and market focus.

These are my 2 cents about AI/NLP. I also have some thoughts on data acquisition and business model waiting to develop.

Quickguide to start a static website with Pelican

2018-09-01T00:00:00-05:00

There are many options to build a website. My reason to use Pelican is because

It's open sourced at github/pelican, with many plugins and themes.
I want to get my website up and running in one week.

If you are familiar with Python and markdown, the Quickstart provides enough instruction to get started. Here I'm putting together the commands and adding some extra plugins for making this website.

Note: Essentially, Pelican converts text files into HTML files, just like pandoc.

TOC

Get started
File structure
Themes
Setups
Developing
Write the first article
Add plugins
Useful tricks

Get started

# installation
pip install pelican markdown

# start a project
mkdir -p ~/projects/yoursite  # make sure there is no special character in the path to your project
cd ~/projects/yoursite
pelican-quickstart

For questions that have default values denoted in brackets, feel free to use the Return key to accept those default values. Don't worry too much. Most of the values are stored in pelicanconf.py and publishconf.py.

File structure

Here is an example of the file structure for a demo project.

.
├── Makefile  # use `make help` for manual, it is very self-descriptive 
├── content
│   ├── articles  # folder for blog articles
│   │   └── hello.md
│   ├── pdfs  # local pdfs to include in articles
│   │   └── file.pdf
│   ├── images  # local images to include in articles
│   │   └── picture.png
│   └── pages  # front pages like About Me, Research
│       ├── Research.md
│       └── About Me.md
├── develop_server.sh  # script to control a local pelican server, i.e. on a RPi
├── fabfile.py  # script to control a remote server by Fabric
├── output  # output of html files
│   ├── ...
│   └── ...
├── pelicanconf.py  # configure file based on answered values
└── publishconf.py  # configure only used in `make publish`

Note: Pelican doesn't recognize HTML files in content/pdfs folder at the moment. Any self-generated HTML files has to be manually copied to the output/pdfs folder.

Themes

Resources
- Themes gallery
- Github collections
- pelican blue: the one I used here.

Note: for many themes, the default index is defined as the list of blog articles.

To deploy a theme:

git clone the theme folder
In pelicanconf.py, add THEME = "path/to/theme"

Or use flag when build a project, pelican -t ~/pelican-themes/theme-name

The basic structure of a theme folder is:

.
├── static
│   ├── css
│   │   ├── pygments.css
│   │   └── style.css
│   ├── images
│   │   └── icons
│   └── js
│       └── comments.js
└── templates
    ├── archives.html
    └── ...

To customize or build a theme:

modify or create new css, js files and html templates.
pelican-themes -i {your_theme_name}

Setups

The complete setting list can be found at here. Many of which are not needed. Here I covered some setups for this website.

The pelicanconf.py file explained:

...
# some general setup
SITEURL = 'http://localhost:8000'  # use the url for test/local server
PATH = 'content'  # path to the folder of contents
STATIC_PATHS = ['images', 'pdfs']  # folders to be copied to output, i.e. `images/picture.png` can be accessed by `siteurl/images/picture.png` in any articles
TIMEZONE = 'America/Chicago'  # based on tz database
MENUITEMS = (('Blogs', SITEURL),)  # add index into site map formed by pages
THEME = "path/to/theme" 
PLUGIN_PATHS = ["path/to/plugin1","path/to/plugin2"]
PLUGINS = ['render_math','deadlinks', 'photos', 'representative_image']  # see `useful plugins` section

# optional setup
TWITTER_USERNAME = ''  # if using twitter related functions
LINKS = (('Pelican', 'http://getpelican.com/'),)  # external links
SIDEBAR_DIGEST = ''
FAVICON = '/images/favicon.ico'  # see note

# dead link options
DEADLINK_OPTS = {
    'archive':  True,
    'classes': ['custom-class1', 'disabled'],
    'labels':   True,
    'timeout_duration_ms': 1000,
    'timeout_is_error':    False,
}

Note: FAVICON is the icon for this website, mine is made of my initials using favicon-generator.

The publishconf.py file explained:

SITEURL = ''  # your web domain
MENUITEMS = (('Blogs', SITEURL),)  # add index into site map formed by pages
RELATIVE_URLS = False  # document-relative URLs, useful when developing

# feeds generation
FEED_ALL_ATOM = 'feeds/all.atom.xml'
CATEGORY_FEED_ATOM = 'feeds/%s.atom.xml'
FEED_DOMAIN = SITEURL

DELETE_OUTPUT_DIRECTORY = True  # delete develop cache if set in True

DISQUS_SITENAME = ""  # see notes
GOOGLE_ANALYTICS = ""  # see notes

Note:

DISQUS: follow guide to use DISQUS for commenting. Static comment plugins exist to have full control of contents.
Google Analytics: Set to ‘UA-XXXX-YYYY’ from Google Analytics to help track traffic.

Developing

It is very straight-forward if you know how to read the Makefile.

Start the server: make devserver or ./develop_server.sh start
Stop the server: make stopserver or ./develop_server.sh stop
Publish the website: make publish
Upload website to web server
- FTP: make ftp_upload
- ssh: make ssh_upload
- using Fabric: python fabfile.py
- ...

Write the first article

Title: My First Article
Date: 2010-12-03 10:20
Category: Articles
Tags: tag1, tag2
Featured_image: /images/picture.png
Status: draft
Modified: 08/14/2018


Article Content In Markdown Format

Add plugins

A collection of plugins can be found here

Use of plugin:

git clone the plugin folder
In pelicanconf.py, define PLUGIN_PATHS and PLUGINs as explained in Setups.
Add additional setup following special instruction of the plugin.

Useful tricks

copyright footer: generate free privacy policy and style it to protect your website
render_math: support math in markdown
representative_image: extracts a representative image (i.e, featured image) from the article's summary or content.
pin_to_top: pin an article to top

PPTP on OS X Just One Last Time

2018-08-14T00:00:00-05:00

TL;DR manual for connecting to PPTP VPN on a MacBook, summarized by Dongming Jin

## 1. create the PPTP configuration
sudo mkdir /etc/ppp/peers && sudo cat <<EOF > /etc/ppp/peers/pptpvpn
plugin PPTP.ppp
noauth
remoteaddress xxx.xxx.xxx.xxx  # address of the VPN server, in my case: 23.24.218.210
redialcount 1
redialtimer 5
idle 1800
mru 1368
mtu 1368
receive-all
novj 0:0
ipcp-accept-local
ipcp-accept-remote
refuse-eap
refuse-pap
refuse-chap-md5
user USERNAME  # account name of the VPN server, in my case: TestUser2
hide-password
mppe-stateless
mppe-128
looplocal
password THEPASSWORD  # the password provided
nodetach
ms-dns 8.8.8.8
ipparam gwvpn
debug  # future diagnose in case of failure
logfile pptp.log
EOF

## 2. start the connection
sudo pppd call pptpvpn

## 3. direct the traffic, this is the ip address of the remote desktop
sudo route add -host xxx.xxx.xxx.xxx -interface ppp0  # in my case: 192.168.51.12

Important Note

OS X has discontinued the support of PPTP since 2016. PPTP VPN is not secure as brute force attack tool exists to decrypt traffic over the VPN tunnel. The most practical advice is to move on to IPSec or SSL VPN.

reference

Future Transportation and Billion-Population Cities

2018-07-22T00:00:00-05:00

Here is my comment to the flying car design by Rolls Royce.

A pricey future. High cost of R&D, city reconstruction and potential failure damage.

You may consider about guided glider with modular on a rail. At least it won’t crash with power failure and when it crashes, no exploration or second damage imposed. Or just us AI to detach and glide to land. Still keeping the benefit of high volume traffic in air and city can then be built vertically with higher population density, which in turn is the use case of such transportation.

A good point to make a public checkpoint public for an idea I have thought of and researched on.

Like this.

And the station like this.

Hit me for more insights about,

fuel efficiency compared to train, car and airplane in different aspect
catch and release techniques
stations and modularities
future million-population buildings
aerodynamics and tunnel experiment, collaborated with a professor from Beihang University and maybe UT Arlington ARC lab.

I named it HATS: high-speed aerial transportation system.

Boarding Glasses for Motion Sickness

2018-07-21T00:00:00-05:00

Boarding Glasses built a pair of glasses filled with colored liquid to act as reference of ground, which can reduce the Motion Sickness.

There must be more patented techniques involved but 99 euro is a bit pricey. So I prototype this one to print and test it out.

Next steps:

find the right the viscosity of the liquid: the liquid should response to acceleration in certain way?
the right shape of the inner ring: circular, oval, or a better functioning shape?
design: is it necessary to have 4 pieces? Maybe this is designed to add on normal glasses?
FOV coverage: optimize the shape and size to set a standard for flexible design.

Here is my knock-off design to reverse engineering the Boarding Ring technique.

Comments on the arising state-sponsored Trolls

2018-07-20T00:00:00-05:00

Bloomberg published an article about state-sponsored trolls in a style. It leads to a report by Future Lab. I'm shocked but also pleased for the public awareness it may bring after the heated discussion about net neutrality few month ago.

Equality is the most treasurable property of the internet. It ensures freedom and unalienable rights, that among these are Life, Liberty and the pursuit of Happiness.

We fear the villains will take advantage to spread crimes to unprecedented wide range of the society but on the other hand, hegemonism and despotism find their way to the world.

Anonymous is the protection that ensures equality at first but now the thread to the security of an opening online society that augments the physical world.

We will not be afraid and together, we will find and perfect the techniques to avoid such sociological scam. Among these are differential privacy, blockchain, open source spirit and humanity! We should rethink about anonymity, which is getting outdated in the flatting world and only brings psychological attraction to behave out of box.

Reach me out about building an identification system that should be passed to the next generations and beyond.

A post first shared here. Right after my comments on Decromacy.

How to ensure Democracy?

Public education & Internet: free and open access to information

Technology & Non-discrimination: net-neutrality, distributed voting system, blockchain, ...

Thoughts about The Nature of Sensing

2018-07-19T00:00:00-05:00

One foundation of the modern technology is advanced sensing, surveying the space in different scales with various fields of physics.

For instance,

light, laser, infrared, radio, X-ray, miniature radar, : essentially these are all EM waves
sound, seismic waves, sonar, ultrasonic: which are mechanic waves
gravitational waves!

...

Each of the subject has extensive research and applications done and ongoing. Many of these will start in lab and then be used in distance measurement (1-D), then be used in imaging (2-D) and ultimately space surveying (3-D).

I am no expert in any of the explicit topics but what I learned from physics is to think in first principle. That's how complexity collapses and technology reveals the nature.

What I do in astrophysics is just another aspect of the same principle. Except we have to use whatever we can and have to squeeze the information out. Because we don't have the energy for active measurement (shout and hear back) in such scale. Data analysis in astronomy tries to utilize every pixel or even the dynamical photon counts (band and wavelets over the time in radio astronomy) to infer the source characteristics.

A post first shared here.

Disrupting Yourself for greater

2018-07-17T00:00:00-05:00

This is what I learned from Video Link.

Outline

Disrupted innovation

Taking the right risks
- Look at what others are not doing: open new doors instead of red seas
Distinctive strength
- Do the job well then focus: principle
- People appreciate but you dismiss: awareness, stay passionate
- Self fulfilled: enthustic
Embracing constraints
- Understand the constraints and head on to it: challenge the edges
- Limit induces creation: change leads to breakthrough
Battling entitlement
- Don't take for granted
- Feel hungry
- Be open to differences
- Appreciate the under-appreciated
Step back to grow
- Step out of comfort zone: be aware not to settle
Fail
- Don't fear: learn and move on
Let your strategy emerge
- Find your metrics: set scale and goal
- Show up, practice: try and improve

WorkFlow and Methodology

2018-07-14T00:00:00-05:00

Table of contents:

Overview
Drafting
- Outline: MindNode
- Extend: export to Markdown
Developing
Publishing

Overview

The following map summarized my general WorkFlow for data analysis based on Python, which is a popular community-driven and user-friendly language.
I hope the WorkFlow and Methodology beneath can also serve as a reference for other language/projects, where Python is found useful as a prototyping/demonstration tool.

This is a work in progress, and will be aggregated into a talk about Open Research. My idea of open research is adopted from the computational thinking from open source community.

About ME

Some of the basic concepts include,

continuous integration: test-driven and object-oriented for easier maintenances.
well documented: including report/web-page/presentation for better understanding.
reproducibility: containerized developing environment, archived data and git-logged code repository should be accessible to replicate.
community driven: public ready to communicate on peer reviews and general feedbacks.

Drafting

Outline: MindNode

Use mind map to conveniently remark and organize the outline of a project.

Content type in MindNode

image
smart link with http://
note
task

Extend: export to Markdown

Export the mind map to markdown document to extend the details on each topic.

Markdown Basics: the key formatting syntax. Markdown is also compatible to use html markup most of the time.

<!-- image with style -->

[<img src="img_url" style="float: right" width="100px">](img_url)

<!-- cross reference -->

TOC

- [title](#tag)
  <a name='#tag'>title</a>

<!-- footnote -->

<sup>[1](#myfootnote1)</sup>
<a name="myfootnote1">1</a>: Footnote content goes here

Developing

Environment: Docker

Containerize: balancing between system isolation and performance like a sandbox for micro-service.

Basic concepts

Images - The blueprints of our application which form the basis of containers. In the demo above, we used the docker pull command to download the busybox image.

Containers - Created from Docker images and run the actual application. We create a container using docker run which we did using the busybox image that we downloaded. A list of running containers can be seen using the docker ps command.
Docker Daemon - The background service running on the host that manages building, running and distributing Docker containers. The daemon is the process that runs in the operating system to which clients talk to.

Docker Client - The command line tool that allows the user to interact with the daemon. More generally, there can be other forms of clients too - such as Kitematic which provide a GUI to the users.
Docker Hub - A registry of Docker images. You can think of the registry as a directory of all available Docker images. If required, one can host their own Docker registries and can use them for pulling images.

1. Start from base image

Basic commands

docker pull image_name  # pull public image/repository from a registry

docker build [--no-cache] -t image_name path/to/Dockerfile [-f renamed-dockerfile]

docker run -it  # interactive
           --rm  # rm container when exit
           -d  # run as detached
           -p 8888:8888  # port fwd to host
           -e DISPLAY=$DISPLAY  # set environment variable
           -u user  # username/uid in image
           -v path/to/local:path/to/container  # mount directory
           image_name
           [command]

docker port container_id  # show the open ports of a container instance
docker start/attach/stop/rm container_id  # manage a container instance
docker rmi image_id  # remove an image

public registry: be caution about potential security risk in 3rd party image/repository.

2. Record needed ingredients in requirement.txt

While developing, record the additional python packages in a text file named requirement.txt, which will be useful to construct the Dockerfile to automatically configure the developing environment, as well as hosting an interactive Jupyter notebook with mybinder.

file format and specifiers
pip freeze > requirement.txt: list installed package

# Example requirement.txt
SomeProject
SomeProject == 1.3
SomeProject >=1.2,<.2.0
SomeProject[foo, bar]
SomeProject~=1.4.2

3. Compose building recipe to Dockerfile

Some resources

Best practices: use minimum system with necessary packages
Base app samples: from IDE to OS, official docker images to start with
Docker Hub: Community hub, use with caution

Steps

start with a base image that meets the need of the project
customize with additional dependency using RUN sudo pip install -r requirement.txt
add special system setup and port configuration as needed

Prototyping: Python

Some principles

Object-oriented programing
function -> method
variable -> attribute
Computational thinking: see the full image here

A few suggestions

think about goals of code
break into reasonable classes
pseudocode it up
check for efficient algorithm
test each function and class
assemble code in main
run
optimize if necessary

Code Styling: PEP8

indentation: 4 spaces
snake_case: packages, modules, functions
CamelCase: classes, attributes
_alltogether: variables, add _ for build-in variables_
space near operators, except for func(a=1, b=3) or c = a/b
separate imports
>2 space # 1 space inline comments, docstring
break liner /

Practical steps

1. Dev with Jupyter, note issues: interactive notebook is very handy at development

Pitfall with interactive notebook
execution order matters
memory consuming
See my LSST talk about Jupyter as a Research Tool for example.

2. Aggregate to python script: modularize codes into functions

Packaging: iPython

When the code is ready to share,

create a parent directory
create setup script
create MANIFEST.in for list of files
include README
include LICENSE
include setup.py
recursive-include folders scripts
python setup.py sdist

1. Modularize function: if not done earlier

class CamelCase():
    def __init__():  # initialize
        self. ...
    def __repr__(self):  # print
        return "..."

if __name__ == '__main__':
    body of program

2. Unittest: remember the issues we note down during the developing? These are good cases to write up tests about. A more proactive concept is test-driven programming.

├── __init__.py
├── code.py
├── func_a.py
├── func_b.py
├── func_c.py
└── tests
    ├── __init__.py
    ├── test_funcs.py
    └── test_something.py

testing framework

pytest
conda install pytest pytest-cov
compose tests under tests/
run py.test
nose

3. Continuous Integration: use continuous integration to automatically test when something changes in repository.

Travis: require .travis.yml file to config
circleci
AppVeyor

# example .travis.yml file

language: python
python:
  - "3.6"
# command to install dependencies
install: "pip install -r requirement.txt"
# command to run tests
script: pytest

4. Profiling & Optimization

Premature optimization is the root of all evil. -- Donald Knuth

computation amount
memory usage
input/output
storage

Tips from Cameron Hummels

Decide what you are optimizing over
Computer time versus person time
Write readable code first, then optimize
Use profilers to identify bottlenecks
Address bottlenecks one at a time
Latest Python is most optimized
Try new approaches and profile/testit

Object oriented code
NumPy arrarys are optimized
Vectorize loops when possible
List comprehensions
Avoid building lists through appends
In place operations as opposed to rebuilding
Cython
Numba

time  # coarse total time
%%timeit  # code snippets
cProfile  # deep profiling with viz tools
  pstats  # text-based
  snakeviz
  runsnakerun  # pipeline tool
  pyprof2html  # html tool
line_profiler/kernprof  # line-by-line function
memory_profiler  # memory consumption and memory leaks

Parallel computing

multithreaded
multiprocessing
MPI and mpi4py

5. Documenting: essential for future revisit or further development.

versioning: x.y.z (E.g., 0.2.3, 2.7.12, 3.6)

• change x for breaking changes • change y for non-breaking changes • change z for bug-fixes

docstring and comments tips

document while coding
document the interfaces of modular
use descriptive names
consistent style/format
docstring

Publishing

Presentation

Following my WorkFlow, most of the work has been done at this stage. The rest can be carried out in very minimum effort with decent finish.

Example finish: use ? for keyboard shortcuts to control the slides.

Jupyter notebook to html report: my Ph.D. project
Jupyter notebook to html slides: my talk slides on LSST 2018 workshop
MindNode to markdown to html slides: my draft slides of this document

1. MindNote -> Markdown: re-arrange and convert the outline mind map to markdown.

Warning: Jupyter notebook should be re-organized for presentation, especially dissertation defense! The order of work is not necessary the order of talk! Check my LSST talk for some tips.

2. Markdown -> HTML: extend the details in markdown and convert to html.

Pandoc: powerful tool for conversion.

install: brew install pandoc
use % for frontpage info
usage demo: pandoc -s --mathjax -i -t --slide-level=2 revealjs WorkFlow.md -o WorkFlow.html

3. Slideshow: HTML + reveal.js

add -V revealjs-url=http://lab.hakim.se/reveal-js when using pandoc to convert
or download reveal.js to the same directory of the converted directory
Now it's ready to open the html file to start the slideshow, use ? for keyboard shortcuts to control the slides.

Workshop

To take the research/project to a workshop, we need to recall what we've done.

1. Config env.: Dockerfile

With everything done, it is now easy to put all the ingredients and recipe together into a Dockerfile.

# Example Dockerfile for python
cat > Dockerfile <<EOF
FROM python:3

WORKDIR /usr/src/app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "python", "./your-daemon-or-script.py" ]
EOF

# build
docker build -t pydev .
docker run -it --rm pydev

2. Demo: Markdown + Scripts -> Jupyter notebook

Following the mind map and resulting markdown file, we can put the outline structure in Jupyter notebook since it natively supports markdown formatted cells. We can fill in the function calls and visualization codes in between.

My secret on toggling code cells

nbextension
codefolding: $ jupyter nbextension enable codefolding/main
more extenstions: spellchecker, Table of Contents, Autoscroll,...
nbconvert with --template template.tpl
hide/remove code in one step templates
more functionality available

Another wheel to edit slide styles on the fly

RISE: good for dev./workshop
demo: interactive live rendering, based on reveal.js
reveal.js: good for prod./presentation
post-process with nbconvert

3. Slideshow: nbconvert + reveal.js

Wrap up commands to convert notebook into slideshow

jupyter nbconvert notebook.ipynb --to slides --reveal-prefix reveal.js [--template hidecode/rmcode.tpl]

wget https://github.com/hakimel/reveal.js/archive/3.6.0.tar.gz | tar -xvzf -

open notebook.slides.html

Repository: GitHub

When it is ready to take the project to the public, there are a few wheels very handy to make it more appealing.

Live slideshow: add some markup in the url of the html file in repository to render the slideshow in live, not always working.

reveal html files: go to http://htmlpreview.github.io/?+git_html_url+?print-pdf

Live notebook demo: binder

Everything is ready, just paste the repository link to mybinder.org

configure: add requirement.txt or Dockerfile
launch on binder: recall the RISE demo
static option: open http://nbviewer.jupyter.org/ and paste the github url of the ipynb file

Project webpage: github.io + HTML

github.io: use any of the converted html file to set it up in 3 steps

create a repository with username.github.io
add HTML file
go to https://username.github.io

Documentation: Read the Docs

This step depends on how often and well the project is documented. If earlier guide is followed, there is no pain at all.

sphinx with reST: sphinx-quickstart -a "Name" -p Repo -v 0.1 --ext-autodoc -q
doxygen
readthedocs: link github & auto build

Building test: Travis CI

Follow the manual/documentation!

tricks

# load dataset based on memory
sources = pd.read_csv(chunksize=100000)
for i, chunk in enumerate(sources):
    ...

Neural Networks

2018-07-05T00:00:00-05:00

Building Blocks

Gradient Descent

finding the optimum in parameter space

Strategies:

Batch: $\theta = \theta - \eta \cdot \nabla_ \theta J(\theta)$
Stochastic: $\theta = \theta - \eta \cdot \nabla_\theta J(\theta;x_i;y_i)$, faster but biased
Mini-batch: $\theta = \theta - \eta \cdot \nabla_\theta J(\theta;x_{i:i+m};y_{i:i+m})$, intermediate

Challenges:

$\eta$: apply learning rate schedules
Saddle points as local minimum: spiking NN for quantum tunneling?

Improvements for SDG:

add momentum: $v_t = \gamma v_t - 1 + \eta \cdot \nabla_\theta J(\theta); \theta = \theta -v_t$, reduce damping
NAG, Nesterov accelerated gradient: add correction based on momentum, $J(\theta) -> J(\theta - \gamma v_t - 1)$, avoid momentum crash
Adagrad, adaptive gradient: $\eta_i$ for each $\theta_i$, $\theta_{t+1}=\theta_t - \frac{\eta}{\sqrt{G_t+\epsilon}}\odot g_t$
Adadelta: reduce the calculation of $G_t$ with a window-based average, $E_t$
RMSprop: $E_t = \gamma E_{t-1} + (1-\gamma)g_t^2$
Adam, adaptive moment estimation: bias-corrected from weighted average mean and variance

Asynchronous SGD for parallelization:

Hogwild
Downpour SGD
Delay-tolerant Algorithms for SGD
TensorFlow
EASGD, elastic Averaging SGD: local variable fluctuate from center variable

Extra to explore:

Shuffling and Curriculum Learning
Batch normalization: keep N(0,1)
Early stopping: avoid overfitting
Gradient noise: shuffle away from local

Overview of Gradient Descent refs

Activation Function

back-propagatable, non-linear projection

keep nonlinearity & non-vanishing gradient

Sigmoid: $f(x) = \frac{1}{1 + e^{-x}}$, work with cross-entropy cost function $y\ln{a}+(1-y)\ln{(1-a)}$ to avoid gradient vanishing
tanh, $tanh(x)= 2 \sigmoid(2x)-1$: better but not the final solution
ReLU, $y=x\ge0? x:0$
- constant gradient on one side
- output shift & hard to converge
PReLU, $f(y_i)= y_i>0? y_i: a_iy_i$: regulate the left side
- He, K., et al. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. ICCV 2015
- RReLU, Xu, B., et al. Empirical Evaluation of Rectified Activations in Convolutional Network. ICML Deep Learning Workshop 2015
Maxout, $\max{w_ix+b_i}$
ELU, $f(x)= x>0?x:\alpha (\exp(x)-1)$
- ImageNet example
Noisy Activation Functions, Gulcehre, C., et al., Noisy Activation Functions, in ICML 2016. 2016
CReLU, pair-grouping phenomenon
MPELU

Practical tips:

ReLU
ELU
PReLU/MPELU with regularizer/penalty

Regularization

minimize the loss function with reasonable complexity

Supervised learning: $w^* = \arg\min_w \sum_i L(y_i, f(x_i;w)) + \lambda \Omega(w)$, solve ill-conditioned matrix

L0/L1, minimize the absolute sum: Lasso regularization, make $w$ sparse, $||w||_1 \le C$, linear bound

Feature Selection: less feature needed
Interpretability: less complex

less features

L2, minimize the square sum: Ridge Regression/Weight Decay, $||w||_2 \le C$, quadratic bound

won't reduce features
but reduce certain feature dependence

Tuning Methods

the evil approach

Neural Networks

Feedforward Neural Network: word2vec
- Huffman tree
- CBOW: words -> word
- skip-gram: word -> words
Denoising Autoencoders: obtain good representation
- perform noise mapping: $x \rightarrow \tilde{ x}$
- keep loss as $\mathcal{L}(x,\tilde{x'})$
Restricted Boltzmann Machine: unsupervised learning
- generative approach: obtain $P(X,Y)$ for P(Y|X); in contrast of discriminative approach, which only cares about $P(Y|X)$
  - methods: Markov process & Gibbs sampling
  - metrics: KL divergence, Shannon entropy, $\sum_i P(i)\log\frac{P(i)}{Q(i)}$
- probability distribution: $P = \frac{1}{Z}e^{-E(v,h)}$
- energy function: $E(v,h) =-v^T Wh -a^Tv -b^Th$, $v$ visible unit, $h$ hidden unit
Generative Adversarial Network: Turing learning
- discriminator: convolutional
- generator: deconvolutional
- applications
  - synthesis face from text: progressive growing of GANs, training method
  - recover features in astrophysical images of galaxies beyond the deconvolution limit
  - EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis
  - generating images in latent space
Residual Neural Network
- plain layer + shortcuts
- residual function: $\mathcal{F}(x):=h(x) - x$
- shortcut: $y=w_s x + \mathcal{F}(x, {w_i})$
Convolutional Neural Network: computer vision
- convolutional layer with kernels
Recurrent Neural Network
- LSTM: a memory cell $c_t$, an input gate $i_t$, an 'output' gate $o_t$ and a forget gate $f_t$. $x_t \rightarrow h_t$
- Attention: potential of memory
- applications
  - Semantic MEDLINE: MEDLINE intelligence, NLP on medical papers
  - ORiGAMI: term weight and so on
  - Acoustic modeling: ASR
  - Image description

Spiking Neural Network (3rd generation)

Spiking neuron: accumulate their activation into a potential over time, and only send out a signal (a "spike") when this potential crosses a threshold and the neuron is reset.

Integrator: memory or nonlinear response

BP on SNN
SNN on very low bit rate speech coding: replace HMM
DL in SNN: low energy cost, but no BP, spike time & spike rate

Refs

To Explore

Google Collaboration Lab for notebook dev

Extras

ethics principles: bias imposed by training data

Quantum Computing

2018-07-01T00:00:00-05:00

Quantum Computing & Neural Networks

basic concepts

Superposition: $\ket{+}, \ket{-}$
Phase: $T$
Bloch Sphere
Decoherence
- Energy relaxation: $T_1 := \ket{1} \rightarrow \ket{0}$
- Dephasing: $T_2$

Resources

IBM Open Quantum

OpenQasm Input

// My First Score
OPENQASM 2.0;
include "qelib1.inc";

// Register declarations
qreg q[2];
creg c[2];

// Quantum Circuit
// Pauli operations 
x q[0];
y q[1];
z q[0];
barrier q;
// Clifford operations
h q;
s q[0];
sdg q[1];
cx q[0],q[1];
barrier q;
// non-Clifford operations
t q[0];
tdg q[1];
barrier q;
// measurement operations
measure q -> c;

# my_first_score.py
from qiskit import QuantumCircuit, QuantumRegister, ClassicalRegister, execute

# Define the Quantum and Classical Registers
q = QuantumRegister(2)
c = ClassicalRegister(2)

# Build the circuit
my_first_score = QuantumCircuit(q, c)
# Pauli operations 
my_first_score.x(q[0])
my_first_score.y(q[1])
my_first_score.z(q[0])
my_first_score.barrier(q)
# Clifford operations
my_first_score.h(q)
my_first_score.s(q[0])
my_first_score.s(q[1]).inverse()
my_first_score.cx(q[0],q[1])
my_first_score.barrier(q)
# non-Clifford operations
my_first_score.t(q[0])
my_first_score.t(q[1]).inverse()
my_first_score.barrier(q)
# measurement operations
my_first_score.measure(q, c)

# Execute the circuit
job = execute(my_first_score, backend = 'local_qasm_simulator', shots=1024)
result = job.result()

# Print the result
print(result.get_counts(my_first_score))

Operations

  X=(01;10), control-not, CNOT gate
  T=(10;0eiπ/4)
  H=1/√2(11;1−1), Hadamard gate
  S=(10;0i):=T^2
  Z=(10;0−1):=T^4
  S†=(10;0−i):=T^6
  T†=(10;0e−iπ/4):=T^7
  Y=(0i−i0):=XZ

Quantum Algorithms

Shor's algorithm: ordering, factoring
- period finding: modular exponential function, $a^r = 1 (\mod N)$
- steps
- pick $a$, compute $\gcd(N,a)$
- if not co-prime
  - do find period $r$ so that $a^r = 1 (\mod N)$
  - until $r$ is even
- check $\gcd(a^{r/2}\pm 1, N)$ for prime factor
- quadratic sieve method $\exp(d^{1/3})$
Grover's algorithm: reflection^n to amplify the matched state
Quantum Annealing

Technicals

Building blocks - Discussion of transistor

Neruoscience

brief about nervous system
neurons: chemical interactions between neurons as communication, not fixed, multi-connected

Ideas

3D transistor to resemble neuro, spiking instead
growing network, mimic brain development, let network layers to change & train
- biological growing: nervous system growth, need to study human baby
  - timeline
  - day 13:
  - embryonic day 42 - midgestation: establishing rudimentary neural networks
  - 3rd gestational week: differentiation of neural progenitor cells
  - 8th GW: rudimentary structures of the brain and central nervous system
  - rapid growth and elaboration
  - end of the prenatal period: major fiber pathways complete
  - before preschool: increases in size by four-fold
  - by age 6: ~ 90% of adult, structural changes continue
  - 100GB neurons, 60TB connections: ~ 600 link per neuron, multiple-in-guided-one-out
- distribution out + spiking activation
- sectioning during training: vision, language, motion and so on. develop as grow, over the cause of infant
  - self-sectioning: train as needed
  - resonance ignition: transfer learning & creativity

SlideShow with reveal.js

2018-06-17T00:00:00-05:00

Presentation

Docker

configure a docker with necessary ingredients

# based on jupyter/base-notebook
docker pull jupyter/base-notebook

# or use Dockerfile, if need root, https://github.com/jupyter/docker-stacks/issues/211
cat << EOF > Dockerfile
# Copyright (C) 2018 by Dongming Jin
# Licensed under the Academic Free License version 3.0
# This program comes with ABSOLUTELY NO WARRANTY.
# You are free to modify and redistribute this code as long
# as you do not remove the above attribution and reasonably
# inform receipients that you have modified the original work

FROM jupyter/base-notebook

MAINTAINER Dongming Jin "dongming.jin@utrgv.edu"

# Switch account to root and adding user accounts and password. or not
USER root
RUN echo "root:Docker!" | chpasswd

USER jovyan
EOF

# build from Dockerfile
docker build -t jupyslides .

sudo for docker

run the container

# test system configuration
docker run -it jupyslides /bin/bash

# load with data and slides
docker run -it -p 10000:8888 --rm -v /Users/domi/Dropbox/Research/Local_universe:/home/jovyan/slides -v /Users/domi/Desktop/LISA:/home/jovyan/LISA jupyslides /bin/bash

# convert to slides
jupyter nbconvert report.ipynb --to slides --reveal-prefix reveal.js --template hidecode.tpl

# replace initialization <script>

Toggle/Remove Code

[in-situ] nbextension to toggle code(metadata level) & more beyond
- doc
- github install: conda install -c conda-forge jupyter_contrib_nbextensions
- configurator
- jupyter nbextension enable codefolding/main
[prod] hide code nbconvert with --template hidecode.tpl, hidecode.tpl
[prod] delete code nbconvert with --template rmcode.tpl
~~HTML buttom~~: need a buttom.., better to use tpl
- <a href="javascript:code_toggle()"> [Toggle Code]</a>
RISE to automatic live reveal
- Usage
- customizing
- simple demo
- shortcuts[editing]
  - alt-r, “Enter/Exit Live Reveal Slideshow”
  - shift-i, Toggle slide
  - shift-o, Toggle-subslide
  - shift-p, Toggle-fragment
  - alt + r: start/stop
  - ,: help
- enable: jupyter-nbextension enable rise --py --user
- diable: jupyter-nbextension disable rise --py --user

Rendering

reveal.js
- github
  - pdf-export
  - transition: None - Fade - Slide - Convex - Concave - Zoom
  - themes: Black (default) - White - League - Sky - Beige - Simple - Serif - Blood - Night - Moon - Solarized
  - full screen background: <section data-background="image.png" data-background-repeat="repeat" data-background-size="100px">
  - background transition: Reveal.configure({ backgroundTransition: 'zoom' })
- change in reveal.js file and rendering with --reveal-prefix reveal.js
  - path for my configure /Users/domi/Desktop/reveal.js
  - brief intro from scratch
- check zoom.js
- shortcut

            'N  ,  SPACE':          'Next slide',
            'P':                    'Previous slide',
            '&#8592;  ,  H':        'Navigate left',
            '&#8594;  ,  L':        'Navigate right',
            '&#8593;  ,  K':        'Navigate up',
            '&#8595;  ,  J':        'Navigate down',
            'Fn + <':                   'First slide',
            'Fn + >':                   'Last slide',
            'B  ,  .':              'Pause',
            'F':                    'Fullscreen',
            'ESC, O':               'Slide overview'

Remark: demo: github doc

nbconvert

set config in notebook, no need when use reveal.js

from traitlets.config.manager import BaseJSONConfigManager
# get path: jupyter --path
path = "/Users/domi/anaconda3/envs/py35/etc/jupyter/nbconfig"
cm = BaseJSONConfigManager(config_dir=path)
cm.update("livereveal", {
              "theme": "serif",
              "transition": "fade",
              "start_slideshow_at": "selected",
})

command

jupyter nbconvert
xxxx.ipynb
--template=nbextensions: use nbextentions
--reveal-prefix path_to_reveal.js: reveal.js, need update initialization
--post serve: local host

Problems and Fix

\mathsrc replace \mathcal
export to pdf: not working
print pdf: not working as well

replace in html

<!--reveal.js initilization-->
<script>
require(
    {
      // it makes sense to wait a little bit when you are loading
      // reveal from a cdn in a slow connection environment
      waitSeconds: 10
    },
    [
      "reveal.js/lib/js/head.min.js",
      "reveal.js/js/reveal.js"
    ],

    function(head, Reveal){

        // Full list of configuration options available here: https://github.com/hakimel/reveal.js#configuration
        Reveal.initialize({
            controls: true,
            progress: true,
            history: true,
            center: true,

            transition: "slide",

            // Optional libraries used to extend on reveal.js
            dependencies: [
                { src: "reveal.js/lib/js/classList.js",
                  condition: function() { return !document.body.classList; } },
                { src: 'reveal.js/plugin/markdown/marked.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } },            
                { src: "reveal.js/plugin/notes/notes.js",
                  async: true,
                  condition: function() { return !!document.body.classList; } },
                { src: 'reveal.js/plugin/highlight/highlight.js', async: true, callback: function() { hljs.initHighlightingOnLoad(); } },
                { src: 'reveal.js/plugin/zoom-js/zoom.js', async: true }, // using zoom.js
            ]
        });

        var update = function(event){
          if(MathJax.Hub.getAllJax(Reveal.getCurrentSlide())){
            MathJax.Hub.Rerender(Reveal.getCurrentSlide());
          }
        };

        Reveal.addEventListener('slidechanged', update);

        function setScrollingSlide() {
            var scroll = true
            if (scroll === true) {
              var h = $('.reveal').height() * 0.95;
              $('section.present').find('section')
                .filter(function() {
                  return $(this).height() > h;
                })
                .css('height', 'calc(95vh)')
                .css('overflow-y', 'scroll')
                .css('margin-top', '20px');
            }
        }

        // check and set the scrolling slide every time the slide change
        Reveal.addEventListener('slidechanged', setScrollingSlide); 
    }
);
</script>

rmcode.tpl

{%- extends 'slides_reveal.tpl' -%}
{% block input %}
{% endblock input %}

hidecode.tpl

{%- extends 'slides_reveal.tpl' -%}

{% block input_group -%}
<div class="input_hidden">
{{ super() }}
</div>
{% endblock input_group %}

{%- block header -%}
{{ super() }}

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>

<style type="text/css">
//div.output_wrapper {
//  margin-top: 0px;
//}
.input_hidden {
  display: none;
//  margin-top: 5px;
}
</style>

<script>
$(document).ready(function(){
  $(".output_wrapper").click(function(){
      $(this).prev('.input_hidden').slideToggle();
  });
})
</script>
{%- endblock header -%}

Docker Commands

2018-05-30T00:00:00-05:00

docker build [--no-cache] -t img_name path/to/Dockerfile  # build image based on dockerfile
docker images  # all images
docker ps [-a]  # show [all] running containers

docker start/attach con_id  # restart excited container
       stop con_id  # stop container
       rm con_id  # rm container
       rmi img_id  # rm image

docker run -d -p 2222:22 <Image Name>

docker run -it  # interactive
           --rm  # rm container when exit, use with caution
           -d  # detached
           -p 8888:80  # port fwd to host
           -e DISPLAY=$DISPLAY  # set environment variable
           -u docker  # username/uid
           -v <data_location>:~/data  # mount data directory
           --name="rdev"  # container name
           ubuntu  # image name
           /bin/bash  # command

docker port con_id  # show port fwd
ssh -p 2222 root@localhost

docker cp local_file con_id:path/to/target  # copy file to container

Images - The blueprints of our application which form the basis of containers. In the demo above, we used the docker pull command to download the busybox image.
Containers - Created from Docker images and run the actual application. We create a container using docker run which we did using the busybox image that we downloaded. A list of running containers can be seen using the docker ps command.
Docker Daemon - The background service running on the host that manages building, running and distributing Docker containers. The daemon is the process that runs in the operating system to which clients talk to.
Docker Client - The command line tool that allows the user to interact with the daemon. More generally, there can be other forms of clients too - such as Kitematic which provide a GUI to the users.
Docker Hub - A registry of Docker images. You can think of the registry as a directory of all available Docker images. If required, one can host their own Docker registries and can use them for pulling images.

Refs - Docker Q&A - container from scratch

Opening Notes for Defense

2018-04-27T00:00:00-05:00

Graduate Study in a Glance

During my Ph.D. study, I've been questioning about, what is re-search.

It seems to me that research is about

looking at something,
over and over,
in all possible aspects.

During these iterations, we may

narrow assumptions,
apply different methods,
collect more data,
and learn new knowledge.
Reproducibility => why I present in this transparent way
so it is informative about what I conclude and how I deliver

Over the past years, I studied about

Globular Clusters, these are
- stellar systems that consist of millions of stars, they are also
- ancient building blocks of galaxies themselves.

They are extremely interesting environments, in a sense that,

the timescale of stellar evolution, becomes comparable to
the timescale to have stellar encounters.

That is, the stars in these dense star clusters don't die alone,

they will influence the nearby-stars over dynamical interactions,
their own stellar evolutions are also altered during such processes.

In a nutshell,

smaller systems like open clusters cannot stay gravitationally-binded
- stars are easily scattered away and they will evaporate before long
- there are also enough materials to power multiple generation of star formations
larger systems like galaxies instead, have strong gravitational potentials
- most of the stars are trapped and move under a quasi-static field
- the system could be described as statistical equilibrium,

Starting from that, I learned about

the larger structure in the local universe and
built up the concept of the cosmic web

I will expand on that by populating globular clusters in the local universe, which is the second chapter in my dissertation.

Then I had this chance to study about numerical simulations.

I spent the 2013 summer in National Astronomical Observatories, Chinese Academy of Science to study about N-body models with Rainer Spurzem, using Newtonian integrator to solve dynamical interactions.
I also [met with Jonathon Downing and] picked up the Monte Carlo code from Mirek Giersz to evaluate the phase-space evolution in a semi-equilibrium system.

Both approaches provide great details about stellar dynamics based on theoretical assumptions.

From the formation mechanism of compact binary systems,
to the observable global effects of the star clusters, different physics and phenomenons are implemented in various scale.

The adaption of

these numerical methods,
in together with high performance computing techniques, push forward the research greatly so that theoretical prospects become
essential to new proposed instrumental projects,
instructional to reveal discoveries buried in observed data.

I reviewed such aspects in the history of GC studies in chapter 3 and introduced my own simulations to represent the natural assembling of GCs in chapter 4.

The major effects considered in GCs and
the detailed modeling are discussed in my dissertation in corresponding sections. I will only briefly demonstrate the key components later.

From those simulations, I could extract any interesting stellar object with its full dynamical history, especially the compact binary systems that could be potentially the gravitational wave sources in different frequency domain.

I spent 2015 summer at Kavli Institute of Astrophysics and Astronomy in Peking University doing data analysis by querying observational databases for variable sources,
I visited National Astronomical Observatories, Chinese Academy of Science again to prototype a transient object detection pipeline for 50cm Binocular Network with Licai Deng and Richard de Grijs.

The mutual benefits supplemented my experience in optical data processing that promoted my understanding on time series data analysis.

I sketched the work flow to estimate the local abundance of GCs in chapter 2. At the same time, I started to focus on the binary black holes for the resulting GW signals. Matt's code based on Peter's work about general relativity was parallelized to carry out the relativistic evolution for the compact binary systems extracted from GCs. A brief introduction about BBH formation mechanisms and GW astronomy could be found in chapter 5. At the time of the first GW detection, my preliminary result presented on MODEST meeting ruled out GC as the origin of the IMBH merger event.

In 2016 summer, I started training in the LSST data science program and

the systematic coverage of statistical background
and conventional data analysis tools enhanced my modeling of the GC populations.

With the super-cluster in Texas Advanced Computing Center, my collection of GC simulations was also enlarged greatly.

More realizations of the same setup were added to make sure the convergence.
Finer-grained parameter space was explored to avoid extremity.
Selection effect and sampling bias had been made aware as well.

From all the above, a clear image appeared

to derive the detection prospects of BBHs,
which are dynamically formed in GCs,
within 30 Mpc volumetric space
for space-borne GW detectors

I followed

the Fixer Informative Matrix analysis from Cutler and
configured the BBH database to work with Matt's code

This part of work is included in chapter 6.

Without further ado, I will walk through the work manuscript-ed in my dissertation in brief and let you question about any part that is particular interesting to your extent.

Checkout my research work.

Data Management @ LSST

2017-10-19T00:00:00-05:00

4/24/17

SQLite with python

Basic operation syntax

import sqlite3
conn = sqlite3.connect("reintro.db") 
cur = conn.cursor() // creat cursor for operations
cur.execute("""create table PetInfo(
                                    Name text, 
                                    Species text,
                                    Age tinyint,
                                    FavoriteFood text
                                    )""")
cur.fetchall() // display the cursor      
conn.commit() // commit change to database
cur.execute("""drop table if exists xxx""") # drop the table if is already exists

SDSS API

from astroquery.sdss import SDSS
SDSS.query_sql("""select top 20 * from PhotoObjAll""")

demonstration of Join:

4/25/17

SQL byYusra AlSayyad

Data Management: building database, storing data

SQL database
Relational Algebra, not good for graph structure
combined -> pandas by Wes McKinney
TOPCAT: Relational database with uncertainty
Indexing: optimize data extracture and slicing
hierarchical triangular mesh, reduce 2D to 1D for GIS indexing

SQL examples for inspiration

CREATE database mysql_demo;
use mysql_demo;
CREATE TABLE Person (
  personId int,
  firstName varchar(255),
  lastName varchar(255),
  phoneNumber char(10));
CREATE TABLE Publication (
  pubId int,
  title varchar(255),
  journalName varchar(255));
INSERT INTO Person VALUES (1, 'Cameron', 'Hummels', '1234567890');
INSERT INTO Person VALUES (2, 'Y', 'A', '1234567890');
INSERT INTO Publication VALUES (101, 'Stuff about stars', 'Apj');
INSERT INTO Publication VALUES (102, 'Stuff about galaxies', 'Apj');
show tables;

CREATE TABLE Authorship ( ! relational dababase
  personId int,
  pubId int);

INSERT INTO Authorship VALUE (1,101);
INSERT INTO Authorship VALUE (1,102);
INSERT INTO Authorship VALUE (2,101);
INSERT INTO Authorship VALUE (2,102);

SELECT pub.*, p.*
  FROM publication pub 
  INNER JOIN Authorship a
  on a.pubId = pub.pubId
  INNER JOIN Person p
  on a.personId = p.personId

CREATE INDEX key On table

SELECT TOP 10
   p.objid,p.ra,p.dec,p.u,p.g,p.r,p.i,p.z,
   p.run, p.rerun, p.camcol, p.field,
   s.specobjid, s.class, s.z as redshift,
   s.plate, s.mjd, s.fiberid
FROM PhotoObj AS p
   JOIN SpecObj AS s ON s.bestobjid = p.objid
WHERE 
   p.u BETWEEN 0 AND 19.6
   AND g BETWEEN 0 AND 20

SELECT top 10 * 
        FROM photoobjall p 
        inner join ROSAT r 
        on p.objid = r.objid
        where (p.cModelFlux_u + p.cModelFlux_g + p.cModelFlux_r + p.cModelFlux_i + p.cModelFlux_z > 10000) 
        and (p.type = 3) 

SELECT pr.*
FROM
    (SELECT p.* 
        FROM photoobjall p JOIN rosat r ON p.objid = r.objid
        WHERE (p.cModelFlux_u + p.cModelFlux_g + p.cModelFlux_r + p.cModelFlux_i + p.cModelFlux_z > 10000) 
        and (p.type = 3)
    ) as pr
        LEFT JOIN specobjall s ON s.bestobjid = pr.objid where s.bestobjid is null

GROUP BY

ORDER BY ASC/DEC

4/26/17

Data Management: extracting data

Scaling up -> scaling out

Hardware Architecture

Parallel RDMS Architecture: shared nothing
Partitioning: spatical indexing
- fast extracting regional data in high dimension
- random distribute data based on hash table to balance load
Shared scanning
Fault tolerance

Software

Spark, HDFS solution
Hadoop Distributed Files System, HDFS
MapReduce
Relation (shcema)
Indexing
Eventually -> database
NoSQL: not only SQL, non-relational (no schema)
MongoDB
DynamoDB
...
Aggregation-oriented database
pros: same type of aggregated data
cons: hard to slice and dice

LSST QSERV

10/19/17

Resources * SQL notebook * SQL course on KhanAcademy

Machine Learning with MLLib and GraphX @ TACC

2017-05-04T00:00:00-05:00

Zhao Zhang

Supervised Learning
Linear Regression
Classification
- Logistic Regression
- Support Vector Machine: good for non-linear classification
Unsupervised Learning
Lower dimension representation
- Principle Component Analysis
Spare representation
- K-Means
- Gaussian Mixture Models
Independent representation
- Principle Component Analysis

Cost Function

Regularization
Maximum Likelihood
KL divergence
cross-entropy

Graph Processing

frameworks
PageRank: direct graph by Google
Pregel
Giraph
GraphLab
GraphX

GraphX

GraphX abstracts a graph with an RDD of vertices and an RDD of edges

graphx from Apache Spark

Connectd Components: org.apache.spark.graphx.lib.connectedComponents
Triangle Counting: org.apache.spark.graphx.lib.triangleCount
Shortest Paths: org.apache.spark.graphx.lib.Shortestpaths

Scala/Spark @ TACC

2017-05-04T00:00:00-05:00

4/20/17

Introduction to Hadoop and Spark

Dr. Weijia Xu, Dr. Ruizhu Huang @ UTA

Hadoop and Spark on Wrangler
Scala/Spark
Data Analysis Using Hadoop/Spark

Webcast Discussion

Big Data v.s. HPC

Traditional
separate data
MPI code to use multiple nodes
MapReduce: programing model, platform for customized computation
move computations to data, reduce data transfer
sequential execute, same as RDBMS
scale out, not scale up
hardware indepedent

Hadoop

distributed data
key-value pair
computation
map instances
reduce instances

support at TACC * hadoop: open source implementation of MapReduce, programming in JAVA but interface to others * zeppelin: similiar to jupyter, hundreds of interpretator * spark, in-memory

Hadoop and Yarn

Procedure

input
splitting
mapping
shuffling
reducing
final result

Wrangler portal

Manage -> create hadoop reservation

VNC job: access VNC at vis.tacc.utexas.edu
check cluster info. and hadoop job status
idev job
manage data in & out
submit Hadoop jobs
test, dev, debug
batch job: submit jobs to YARN resource manager
large analysis job
sequentially jobs

# manage
## check reservation
scontrol show reservation

## load module and idev to compute node
module load hadoop-paths
idev –r hadoop+TRAINING-HPC+2186 –m 240 –p hadoop  # -m for minutes

## file system operations
hadoop fs -ls/mkdir/put/get/stat/cat/tail/setrep #set replication factor

## YARN, run jobs
yarn jar / hadoop jar # for java
yarn application -list/kill/appStates/appTypes
     node -list
     logs

## upload data
hadoop fs -put local_data data

## running hadoop
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce- examples.jar \
wordcount \ #java class name to run
-D mapred.map.tasks=500 \ #number of mapper instance 
-D mapred.reduce.tasks=256 \ # number of reducer instance
/tmp/data/enwiki-20120104-pages-articles.xml \ #input file on hdfs
wiki_wc #folder to store the output

## get results
hadoop fs –get /tmp/wiki_wc wiki_wc

Hadoop API

Check example at /Users/domi/Dropbox/Open_Course/TACC webinars/Hadoop_Spark/hadoop-training

# recap of hadoop streaming
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar
-input /path/to/input/in/hdfs
-output /path/to/output/in/hdfs
-D mapred.map.tasks=500 \ #number of mapper instance 
-D mapred.reduce.tasks=256 \ # number of reducer instance
-mapper map_implementation  ## can use other languages
-reducer reduce_implementation
-file map system
-file reduce system.

vs Spark

Spark is faster
more language support
MLib for Machine Learning

Zeppelin

web-based notebook
interactive
visualization

# Zepplelin
cp /data/apps/.zeppelin/job.zeppelin .
sbatch --reservation=hadoop+TRAINING-HPC+2186 job.zeppelin

web-UI

%spark

# with Python
%pyspark

# with R
%spark.r

%sh

Contacts

Data Group @ TACC Weijia Xu Ruizhu Huang

04/27/2017

Programming with Scala/Spark

Zhao Zhang

Wrangler Setup

idev -r hadoop+TRAINING-HPC+2187 -t 240 
export PATH=/opt/apps/scala/scala-2.11.8/bin:$PATH

Scala

Google Scala API

Variables
Mutable, var
Immutable, val: cannot be reassigned
Primitive types
Double, Float, Long, Int, Short, Byte: val a: Double = 5
Char, Boolean, Unit
Composite variables
List, Map, Seq, Set, Tuple
- List: immutable for length and element; mutable counter-structure: val l = new ListBuffer[Int/Double]()
- (), head, last, length, reverse, sorted
- Tuple: val t = (1,2,3), t._1 > Int = 1, val(i,j,k) = t
String
Function
(x: Int) => {\n println(x) \n x * 2 \n}
map((x: Int)/x => x * 2) or map(_ * 2)
multiple parameters: x: (Int, Int) => x._1 + x._1 or case (x,y) => x+y
multiple return: (x: Int, y: Int) => (x+3, y+5)
name function: def func(x: Int): Int = x * x as (x: Int) => x * x
Control Flow
Loop
- while ( i < l.length ) {}
- for ( i <- l ), <-
- l.foreach(x => xxx ), =>
- l.map(x => ...)
If ... else ...: l.foreach(x => {if (x%2 == 1) println(x)})
Pattern matching: l.foreach(x => x%2 match {case 1 => println(x) case_ => })
Iterator

## compile
scalac xxx.scala
## execute
scala xxx

## variable vs value
val l = List(1,2,3)  # value, immutable
var l = List(1,2,3)  # variable, can be reassigned

## mutable pointer
import scala.collection.mutable.ListBuffer
var l = new ListBuffer[Any]()
l += 1

## for-loop
val r = l.map((x:Int) => x*2)  # type: Int, Float, Double, String, List[_], ...
val r = l.foreach(x => x%2 match{
  case 1 => println(x)
  case _ =>
})

val l = List(1,2,3,4,5,6)
val i = l.toIterator  # i is a pointer to the List
i.hasNext
i.next
i.toList  # reverse the rest to list

Thread.sleep(1000)

RDD: Resilient Distributed Dataset

immutable, partitioned collection of elements that can be operated on in parallel Programming Models * Transformations * map: val r = l.map(x => x*2), List(2,4,6,8,10) * filter: val r = l.filter(x => x%2 == 0), List(2,4) * groupBy: val r = l.groupBy(x => x%2), Map(1 -> List(1, 3, 5), 0 -> List(2, 4)) * textFile: val lines = sc.textFile(“path-to-file”) * binaryFiles: val rdd = sc.binaryFiles(“path-to-file”) * Actions * count: val r = l.count(x => x%2 == 0), 2 * collect * take * reduce * saveAsTextFiles

Using Scala/Spark

Weijia Xu

Start zepplin on Wrangler: sbatch --reservation hadoop+TRAINING-HPC+2188 -A TRAINING-HPC /data/apps/.zeppelin/job.zeppelin.work -t 240

Data APIs

RDD: Resilient Distributed Dataset
rdd.map(x => x*2)
rdd.reduce(_+_)
rdd.filter(_%3 == 0)
DataFrame: since 1.3
Abstract API, on top of RDD
Schema
off-heap storage, both memory & hard drive
DataSet: since 1.6
Spark 2.x: dataframe = dataset[row]

RDD vs DataFrame vs DataSet

val rdd =  sc.parallelize(0 until 100)
# Conversion
## DF to RDD
val car_rdd = df.rdd
## RDD to DF
val rdd_df = rdd.toDF
## RDD to DS
val rdd_ds = rdd.toDS

# Functions
df.show(int n)
df.printSchema  # types and keys
df.describe(cols)

rdd.filter(_ < 10).collect
rdd_df.filter("value < 10").show
rdd_ds.filter("value < 10").show
rdd_ds.filter(_ < 10).show

rdd.map(_ * 2).collect
rdd_ds.map(_ * 2).show
rdd_df.map(_ * 2).show // will not work
rdd_df.select('value * 2).show

# Spark SQL
df.select($"model", $"mpg" * 1.6).filter("mpg > 4").show
df.groupBy("mpg").count().show
df.createOrReplaceTempView("cars")
spark.sql("SELECT col1, col2, ...
           FROM table1, table2, ...
           [WHERE condition1, AND|OR condition2 ...]
           [GROUPBY col1, ...]
           [ORDERBY col1, ...]")

IO in Spark

File format
JSON: schema
parquet, ORC: JSON with compression
path prefix
default: hdfs:/// -> /tmp/data
specify: file:///
file/folder
automatically read all files within directory
use directory to separate data
existing file: mode(error/append/overwrite/ignore), default is error

# Read CSV file
val df = spark.read.format(“csv”) .options(“header”, true)
.load(“/tmp/data/mtcars.csv”)
df.show()

# More Files
df.write.json("cars.json")
df.write.parquet("cars.parquet")
df.write.option("delimiter","\t").csv("cars.tab")
val df_json = spark.read.json("cars.json")
val df_parquet = spark.read.parquet("cars.parquet")

Workflow

prepare datasets
val cars = spark.read.format("csv").option(...).load("path_to_file").selectExpr("mpg + 0.0 as mpg", "cyl + 0.0 as label"): easy convert to double
val training = cars.sample(false, fraction)
val test = cars.except(training)
assemble feature
val assembler = new VectorAssembler().setInputCols(Array('key1', ...)).setOutputCol('features')
val lr = new LogisticRegression().setMaxIter(10).setRegParam(0.2).setElasticNetParam(0.0)
define analysis
val pipeline = new Pipeline().setStages(Array(assembler, lr))
run analysis
val lrModel = pipeline.fit(training)
results
result = lrModel.transform(test).select('model, 'lable, 'prediction)

Spark Architecture * Create RDDs * HadoopRDD: val lines = sc.textFile("hdfs://names") * MapPartitionsRDD: val kvp = lines.map(name => (name(0), name)) * ShuffledRDD: val groups = kvp.groupByKey() * MapPartitionsRDD: val res = groups.mapvalues(names => names.toSet.size) * collect() * RDD Dependency * Narrow: One To One * Shuffle Dependency: * DAG(Directed Acyclic Graph) Generation * stage 0: HadoopRDD, MapPartitionsRDD * stage 1: ShuffledRDD, MapPartitionsRDD * Schedule Tasks * split each stage into tasks based on partition * reversed order, recursively find parent stages * blocking between stages

delay scheduling

spark.locality.wait (default 3s)
spark.locality.wait.process
spark.locality.wait.node
spark.locality.wait.rack

Executor Deployment
Node
Process: share physical node
Thread: share physical memory

Spark YARN mode, in conf/spark-env.sh

SPARK_EXECUTOR_INSTANCES (default 2) # Process SPARK_EXECUTOR_CORES (default 1) # Thread
Memory Management
Reserved: 300MB, default
User Memory: 25%, user data structure in RDD (value/variable)
Execution: internal storage, shuffle buffer on the mapper
- export SPARK_WORKER_MEMORY 96G
Storage: cache, dynamic, default 0.5 of 75%
- spark.memory.storageFraction in conf/spark-defaults.conf
Increase parallelism
- more reducers by setting spark.default.parallelism
- set number of partitions of the largest parent RDD sc.textFile(path,128)
- YARN: use flag --executor-memory 2g

Scale Computing on Clusters

2017-04-28T00:00:00-05:00

Configure & utilize HPCs

Nirav Merchant @CyVerseOrg

computational thinking
4A: Abstraction, Automation, Ability and Audacity
establish and manage data driven collaborations at global scale
efficient and coordinated use of CI resources: NSF XSEDE, iPlant, campus HPC and high bandwidth
adopting best practices: HEP, Life science
community driven, self-provisioning, extensible and open source: CISE communities
NSF Infrastructure
data rich and knowledge poor
NSF XSEDE $121M every 5 year
- support team
- travel support
Jetstream @ U Indiana
- cmd line, GUI
- DOI, share VMs and then store, publish via IU Scholarworks
- 1-44 cpu, OpenStack
- VMs
- Atmosphere web interface
- direct API access via OpenStack cmd line or Horizon
- admin/root
- allocations
- CV for PI, abstract
- Main description
Technology landscape
cloud
- image: instance (isolated)
- automation
containers
- app, bins, libs
- user-end
- docker: docker engine/OS
- singularity
Toolbox
- Ansible for automation Ansible: playbooks, configuration management, deployment, and orchestration
- Docker for execution environment
- Makeflow workqueue for task distribution makeflow
- Pegasus
  
  High-constrast imaging in the cloud with klipReduce and Findr

Jetstream demo

jetstream homepage train56: 091Z67OEe8jE

Create Project
create image(instance)
visibility
excluding folders
build
launch image

docker run -it --rm -p 8888:8888 -v /home/train56:/home/jovyan/work/ jupyter/datascience-notebook

-it: interactive
--rm: clean after done
-v: volumn

Demand * open innovation, science and collaboration * complexity of infrastructure * evolving technology landscape * data/metadata * extreme information technology -> renew computational platform every x years

Docker

docker run -it -p 8888:8888 tensorflow/tensorflow tensorflow docker

singularity: prepare and configure HPCs

Spark @ Amazon

Industrial * databricks * myria * Daytona sort contest * gartner: hyper cycle * 2012 * 2014 * 2016

Techniques * MapReduce-based * HPC cluster computing * Databases * Transactional * OLTP * Analytic (NoSQL) * OLAP * Latency * CAP Theorem

Partioned data

Availability Consistency

hadoop REDUNDANCY: default 3, for fault torelance

Amazon demo

AWS
S3: distributed storage
EMR: hadoop framework
- create cluster
- hadoop, HBase, Presto, Spark
- instance
- type m4.large
- number: 1 master and x core nodes
- inbound: allow ip
- Zeppelin: notebook format json

%md
%pyspark
sc.parallelize(data, 8)  # `from pyspark.context import SparkContest as sc` in python

RDD.count()/take(n)/takeSample()/map()/collect()
RDD.flatmap()  # tuble to list
RDD.reduce(add)
RDD.repartition(n)
RDD.groupByKey()
RDD.reduceByKey(add)

sc.textFile('s3://path_to_file')

%sql
$var  # user-defined paramter

LSST, avoid lambda function Spark: lazy, not computing until request data

XSEDE by TACC

Resources
HPC
ECSS: Extended Collaborative Support Service
Science Gateways

globus connect

MPI Foundations @ TACC

2017-04-14T00:00:00-05:00

Contact: Victor Eijkhout

Res:

Terminology

Socket: the processor chip
Processor: we don’t use that word
Core: one instruction-stream processing unit
Process: preferred terminology in talking about MPI.
SPMD model: simple program multiple data

MPI

prototype: declarnation

Collectives

Gathering
reduction: reduce, n to 1 MPI_Reduce
gather: collect, subset to one set
Spreading
broadcast: identical, 1 to n
scatter: subsetting MPI_Scatterv

int MPI_Reduce(
    void* send_data, # use buffer, &x
    void* recv_data,
    int count, # size of x
    MPI_Datatype datatype,
    MPI_Op op,
    int root, # not needed for MPI_Allreduce
    MPI_Comm communicator)

int MPI_Gather(
   void *sendbuf, int sendcnt, MPI_Datatype sendtype,
   void *recvbuf, int recvcnt, MPI_Datatype recvtype,
   int root, MPI_Comm comm)

int MPI_Scatter
   (void* sendbuf, int sendcount, MPI_Datatype sendtype,
    void* recvbuf, int recvcount, MPI_Datatype recvtype,
    int root, MPI_Comm comm)

MPI_MAX - Returns the maximum element.

MPI_MIN - Returns the minimum element.

MPI_SUM - Sums the elements.

MPI_PROD - Multiplies all elements.

MPI_LAND - Performs a logical and across the elements.

MPI_LOR - Performs a logical or across the elements.

MPI_BAND - Performs a bitwise and across the bits of the elements.

MPI_BOR - Performs a bitwise or across the bits of the elements.

MPI_MAXLOC - Returns the maximum value and the rank of the process that owns it.

MPI_MINLOC - Returns the minimum value and the rank of the process that owns it.

buffer

void pointer: memory address of the data

C

write &x or (void*) &x for scalar

Python

comm.recv, slow but python object; comm.Recv, fast

find max random number with Reduce

Scan or 'parallel prefix'

Barrier: Synchronize procs

almost never needed
only for timing

Naive realization

root to all $\alpha + \beta*n$
better, p2p

Distributed data

local_index + rank
global_index

Load balancing

$f(i) = \floor(iN/p)$ given proc $i$: $[f(i),f(i+1)]$

Local info. exchange

Matrices in parallel: distribute domain, not the matrix

p2p: ping-pong * two side: A & B * match: send & recv

int MPI_Send(
  const void* buf, int count, MPI_Datatype datatype,
  int dest, int tag, MPI_Comm comm)

Semantics:
IN buf: initial address of send buffer (choice)
IN count: number of elements in send buffer (non-negative integer)
IN datatype: datatype of each send buffer element (handle)
IN dest: rank of destination (integer)
IN tag: message tag (integer)
IN comm: communicator (handle)

send & recv

communication across nodes is x100 slower than within nodes

Blocking

send & recv are blocking operations

deadlock

receive
send

might work

send
receive

Bypass blocking

odds & evens: exchange sort, compare-and-swap
pairwise exchange

Non-blocking

isend, irecv

need MPI_Wait as blocking to refresh the buffer; buffer is safe in blocking comm. Latency hiding: don't need to hold until comm is done TEST: non-blacking wait, do local work if no incoming msg by test

MPI_Bsend, MPI_Ibsend: buffered send

MPI_Ssend, MPI_Issend: synchronous send

MPI_Rsend, MPI_Irsend: ready send

One-sided communication: ‘just’ put/get the data somewhere
Derived data types: send strided/irregular/inhomogeneous data
Sub-communicators: work with subsets of MPI_COMM_WORLD
I/O: efficient file operations
Non-blocking collectives

Data Management @ TACC

2017-04-06T00:00:00-05:00

data@tacc.utexas.edu

by Chris Jordan ctjordan@tacc.utexas.edu

Webcast

Discussion

Slides

Life cycle of the data

Generation for specific purposes
Creation of metadata
Direct use in research/experimentation
Selection and Publication of data
Retirement of inaccurate/outmoded data
Archival of not immediately useful data
Long-term preservation
Incorporation into larger repositories

Open Access, multi-collection repositories

Collection Structure

dated, type doc
consistent in structure
linkage to background/foreground
internal conventions, i.e. norms
Data Management Plan

Data Type

Structured

SQL/RDBMS
semi-structured, HTML/XML

Unstructured

NoSQL
images, audio, mesh data...

HDF5 - Hierarchical Data Formats

self-description
inclusion of metadata
coherence in storing output data

Metadata

metadata: data about data, header

Provenance: "audit trail" for research
Reproducibility
Observational data: date/time/sensor...
Simulational: software, hardware, exec. params.
crucial to record AEarlyAP
automate extraction/creation
enforce good recording

Archiving Research Data

Repositories
- centralized
Accessibility
- maintainace and performance
Evolving
- flexibility to configure
- seamless transition
- long-term agreement
Publish-ability
- scale of data
Preparation
- map of data
- mapping as processing
Execution
- follow practices & permission
- collective note/database/workflow
- batch data management scripts
- cron to automate permissions & movement
- use logs/job output

multiple backup
don't use personal hard drive
don't use commercial "cloud" as primary data store

Workflow Management tutorials

TACC Stockyard

$work: 1TB/2TB by request

project-term: 1 year storage

$scratch: computational work

$home: personal files

Corral: 5TB free, open web access, persistent DNS/Web for long-lived URLs, VMs as backend

http://www.fishesoftexas.org
http://arctos.database.museum

Ranch: 160PB for personal Archive

XSEDE/Wranger: 500TB flash HP storage

GridFTP/Globus Online: web-based, graphical interfaces, useful for XSEDE

iRODS: APIS, web/GUI/cmd, automated enforcement

Get Help

Bash Commands

2017-01-18T00:00:00-06:00

Practical notes for future references.

TeX on OS X

install the BasicTeX from MacTeX
GUI full application: TeXshop
lightweight math validator: LaTeXit

# configure
which pdftex
# install style fiel
tlmgr install

basic Encrypt & Decrypt

openssl enc -in foo.bar[.enc] \
                     [-d] -aes-256-cbc \
                     -pass stdin \
                     > foo.bar.enc

system

scheduling

# backend
crontab -e  ... # m h dom mon dow  command,  2 7-18 * * 1-5  do something
# in-situ
watch -n 5  ... # every 5 sec

info

df -h  # disk storage
du -h  # file size
tree  # directory in tree

updatedb/locate
which/whereis

-manage

purge  # memory
chmod  # 4: r--; 5 r-x; 6 rw-; 7 rwx

for file in `ls *`; do echo $file; done  # ``: prior execution
find . -type f -print -name *.bak | xargs rm  # xargs: streamline

Ctrl-Z  # push to background
jobs
fg  # bring back
kill -STOP/-CONT 

wget  # -nd: no directories; -e exclude_file; -A file_type

special characters: # ; . \ / * ? ${} {a..z} $[ ...] & |

files

# stdin
sort/uniqu [-c] 
diff
less [-S]

cat << EOF >> file
xxx
xxx
EOF

cat file | sudo tee [-a] newfile 

sed [-i.bak] 's/foo/bar/g' file  [> newfile]
tr '\r' '\n'  # translate characters
              # \a    <alert character>
              # \b    <backspace>
              # \f    <form-feed>
              # \n    <newline>
              # \r    <carriage return>
              # \t    <tab>
              # \v    <vertical tab>

# interactive
nano/emacs
vi  # 1G/G: top/end; crtl-f/b: forward/backward

manipulating

grep key file
awk 'BEGIN {} {print substr($m,n,p)} END {print FNR}' file

echo t1/ t2/ t3/ | xargs -n 1 cp (-r) xxx  # copy files into one place
echo operate-file | xargs -n 1 cp -r target position
echo $string | cut -d’-’ -f 1

qlmanage -t -s 200 -o . weixin-brands.svg  # generate thumbnails, convert svg to png

compress/decompress

tar -xvzf / -cvzf
zip/unzip/gzip/gunzip

sync & exchange

scp -p port 
rsync --rsh='ssh -p port' -av --progress -partial

Cluster & Special systems - Laohu

bqueues -l
bjobs -l job_id
bkill
btop
busers

TACC

squeue -u domij
sbatch
showq -U domij
scancel job_id
idev

cdw/cds

user guides

CHIP Projects

2016-03-05T00:00:00-06:00

Ideas

[x] BLE
Rocket relay
Flight Radar -need extra antenna
- sudo service fr24feed start
- interface: http://192.168.3.13:8754/settings.html
Autodesk Ember printer
[x] RepetierServer

# Start server
/etc/init.d/RepetierServer start
# Stop server
/etc/init.d/RepetierServer stop
# Restart server
/etc/init.d/RepetierServer restart

RTL-SDR
[x] GNU RADIO: for signal analysis, not enough cpu for heavy task
- Conference
- docs
- gnuradio-companion
- uhd_fft
Spider
Django
[x] Apache server
- web controlled LED sudo service apache2 start sudo apachectl start
[x] WiFi extender
Network filter (AD blocker)
Mass surveillance network
wifi sniffer
Anti surveillance

Useful Links:

Headless Server

USB connect

screen /dev/tty.usbmodem1413 115200
export TERM=ansi  # correct encoding

cu -l /dev/tty.usbmodem1413 -s 115200
~.  # to escape

system admin

usermod -l new old  # change username, check man for more

sudo usermod -a -G i2c chip  # use i2c tools without sudo for chip

sudo shutdown now

ip addr show wlan0

ps -p$$ -ocmd= # check shell

apt-get update/upgrade

apt-cache search keyword

sudo apt-get install sysstat

mpstat/vmstat/netstat/instat

htop 

last

updatedb # for locate database 

virtualenv venv

deactivate

stty < /dev/tty*  # check band rate

update-rc.d -f xxx remove # remove service

uname -a

Install GUI

apt-get install lxde-core

startlxde

Hardware

power management: check temperature

axp209 /etc/axp209

Wi-Fi: Ref.

sudo nmtui

nmcli device wifi list

sudo nmcli device wifi connect '(your wifi network name/SSID)' password '(your wifi password)' ifname wlan0 

nmcli device status

Bluetooth

sudo hcitool lescan  # search BLE device

bluetoothctl
     power on
     scan on
     pair
     agent on
     connect

GPU?
Onboard Power LED

sudo i2cset -f -y 0 0x34 0x93 0x0 # OFF

sudo i2cset -f -y 0 0x34 0x93 0x1 # ON

GPIO
- Pin number
- library